OCS 2007 R2 and Server 2008 R2

 

Concerning supportability of OCS 2007 R2 and Server 2008 R2 hosts and Server 2008 R2 Active Directory domains…

MS KB 982020  and MS KB 982021 outline what it takes to have OCS 2007 R2 operate successfully with Server 2008 R2 AD and to be hosted by 2008 R2 servers.

DCOM error 10009 and certificate requests

I spent the entire day learning and relearning things about ISA, TMG, and Windows Firewall.  This took me way more time than I wanted which highlights the three basic troubleshooting starting points.  I chose the wrong starting point, and then I chose the next starting point wrongly also.  Of the three choices, of course it was the last choice (midpoint and work towards one end) that proved to be the winner.  Turns out that TMG/ISA/WF all had a part in this.  WF was the culprit on the CA, because there is a rule that needs enabling and by default it is not.  TMG/ISA was also an issue, because of the way RPC traffic is filtered.  Even if you have a wide-open rule that says “let everything go from spot A to spot B” RPC traffic gets hosed.  Argh.

Suffice it to say, I have finally resolved my issue, and an ISA/TMG specialist will probably think I am a loser, but WTH, here goes.

The problem was that I could not request a cert from the OCS R2 MMC.  The CA was across a VPN (TMG/ISA).  First little bit was finding out that, for security, the Windows Firewall disables COM+ access by default.  Now, you would think, that following the MSFT paradigm of enabling Windows Firewall rules when the feature/role/application is installed, that installing a CA on the server would turn this on, but noooOOOOOoooo!

You have to enable this rule - in my case this was a ws08r2 DC with the EntCA installed on it.  You don’t need to do this on the requestor, but the target of the request.  So, my OCS FE was requesting a cert from the CA on the DC.  Therefore, on the CA, enable the Com+Network Access (DCOM-In) rule:

Then, because of the VPN link involved, on BOTH sides of the link, the rule that allows the traffic from the internal to the VPN target needs to have the RPC traffic filter disabled.  There is a GREAT article here that I found that finally resolved my issue.  This is the piece of that article that did it for me:

Problem: When you request a certificate using the Certificate MMC snap-in, the request fails. This occurs even if the CA is started and you have sufficient permissions to request a certificate.

Workaround: This issue occurs because DCOM is required to acquire a certificate (this issue also occurs if you are using CA Web enrollment).

• If ISA Server is requesting the certificate, disable the "Enforce strict RPC compliance setting" on the system policy rule. To do this, on the Firewall Policy tab of ISA Server Management, click Edit System Policy on the Tasks tab. Select the Active Directory group in the Configuration Groups list. On the General tab, clear the Enforce strict RPC compliance checkbox.
• If an internal host is requesting the certificate from another network through ISA Server, do the following: in the Firewall Policy tab of ISA Server Management, right-click the access rule allowing the traffic, and then click Configure RPC protocol. On the Protocol tab, clear Enforce strict RPC compliance.

What that means visually on the TMG/ISA side :

I hope this helps you in some small way.  Now, back to the job I was supposed to get done this morning!