Notes from the Lab

Determining Exchange IS TCP Port Number

Add comment October 17th, 2006

I have already described how it can be done using rpcdump tool. The problem with that method however is that an average Exchange server has about 200 RPC interfaces, so the rpcdump output is going to be quite big. That is why we created a new tool called mapirpc. It only shows Exchange IS RPC interfaces. The tool can be downloaded here. The second tool there is called zpoltestdc. It does the same tests that MS policytest, except it tests the DC specified in the command line. The link requires a free registration but it is quite harmless.

The Mystery of Heuristics

Add comment July 17th, 2006

Microsoft would’ve made my job a lot easier if they had documented heuristics attribute. It contains a lot of interesting system information about an Exchange server object for any software that can read from Active Directory. Unfortunately, all the information there is presented in bits and all the documentation… is not presented at all. I take that back. You can find this:

Contains special connector attributes, such as “allow system messages”.

OK, that helps… Also, you can find this:

Bit	Value	Significance
0	0	Replication of attribute between sites
	1	No replication of attribute between sites
1	0	Not accessible by anonymous clients through LDAP
	1	Accessible to both anonymous and authenticated LDAP clients
2	0	Not accessible by authenticated clients through LDAP
	1	Accessible to authenticated clients but not to anonymous clients
3	0	Not an operational attribute
	1	An operational attribute
4	0	Not visible through the Administrator program, on the Attributes tab of the DS Site Configuration object (property page can be used to configure the non-operational attributes of the site)
	1	Visible through the Administrator program, on the Attributes tab of the DS Site Configuration object

Not a lot. Especially, if you remember that the second piece is actually about Exchange 5.5.

During the last several month we also found a couple of things about heuristics information:

• Bit 11 (2048) is set to 1 if the Exchange server is a cluster EVS.
• Bit 29 (536870912) is set to 1 if RPC over HTTPS is enabled. This one is a courtesy of Bharat.

That’s it. If you know anything else on the topic, feel free to contact me.

Perfmon may show incorrect (lower) connections number for POP3 and IMAP4

Add comment June 23rd, 2006

There is an interesting Exchange performance counter bug.

When a client connects to the secure POP3/IMAP4 port and doesn’t do an SSL handshake (for instance, if you just do telnet to the port), Perfmon does not register the connection, i.e. Connections Current counter does not change. But when that connection is dropped, Perfmon registers that event, i.e. Connections Current counter value decreases by one. That causes the counter to show incorrect (lower or ridiculously large if drops below 0) connections number.

At the same time, connections number limit is not affected. For instance, if the limit is set to 100 and the server has 100 connections, the next connection attempt will fail even if the Connections Current counter shows 98 (for instance).

How to reproduce the problem

On the Exchange server watch the following performance counter: MSExchangePOP3(1)\Connections Current.
Initial Connections Current = 0

1. On a client machine, run telnet 110. Connections Current = 1.
2. On a client machine, run telnet 995. Connections Current = 1.
3. On a client machine, close the second telnet window. Connections Current = 0.
4. On a client machine, close the first telnet window. Connections Current = 4294967295.

This test was performed on Windows 2003/Exchange 2003, but I’m pretty sure Exchange 2000 has the same bug. Also, I suspect it affects OWA.

Exchange 2007 and the Old Routing Group Question

Add comment June 14th, 2006

In Exchange 2007, Microsoft (finally!) is getting rid of Routing Groups. Now it’s official. Exchange 2007 doesn’t have its own topology anymore. It fully relies on AD topology. I repeat, it doesn’t just use AD to store its topology, but actually uses AD topology.

So far I heard two kinds of comments about this particular aspect of Exchange 2007. The first kind: “Now we need to make sure our AD sites are configured correctly”. Duh! Newsflash: Exchange is not the only reason your AD topology must be configured correctly.

The second comment is actually more interesting: “Microsoft is finally backtracking out of something they shouldn’t have done in the first place. And they are making a big deal out of it”. While I agree that Routing Groups are a nuisance, I still think we needed them. As far as I know, Microsoft’s standard reasoning is: “We needed Routing Groups to maintain backward compatibility with Exchange 5.5 Sites”. I agree with that. But I think even Microsoft minds don’t always realize how deep that statement really is. I’m pretty sure they are talking about technical compatibility. But there is one more aspect: human resources. I don’t know how many of you remember 1999, but I do remember it pretty well. At that time, I was a Windows NT/Exchange Consultant. For those of you who remember 1999, that last sentence sounded pretty weird. And that’s kind of my point. How many of “Exchange Engineers/Administrators” new Windows NT well in 1999? And what about knowing Windows NT domains, security etc? “Exchange Administrator” and “Windows Administrator” used to be two clearly distinct professions and two distinctly different team in any large company. The line is much blurrier nowadays. Exchange people new almost nothing about Windows NT domains, security etc. If Microsoft had thrown them into AD without their own directory, that would’ve been an HR disaster of global proportions. The level of Exchange 2000 integration with AD created enough problems for Exchange Engineers. They had to learn Windows domains (and by proxy Windows) and DNS and they had to do it fast. And they had to learn to interact with Windows support teams much closer. But Exchange still had its own mailflow topology.

I think you got my point by now. Microsoft was not the only one that needed Routing Groups. Exchange professionals needed them too. Routing Groups allowed Exchange professionals time to be somewhat gradually introduced into Windows directory. And yes, I know that “gradually” in the previous sentence is a huge overstatement of the “graduality” of that whole process.

Exchange 2007: Mailflow Changes

Add comment June 13th, 2006

Here is what I understand so far.

No Routing Groups, No Link State, No Connectors

There is no Routing Group in Exchange 2007. To some degree that functionality has been replaced by AD site. Essentially, Exchange 2007 doesn’t have its own topology. It uses AD Topology to calculate routes, connection costs etc.

Send It Directly

In Exchange 2003, the only case when mail gets sent directly from one server to another is when they are in the same Routing Group. Between Routing Groups, they use Bridgeheads. Furthermore, if there is no direct Routing Group Connectors, several Bridgeheads get involved. All of this stops with Exchange 2007. By default, every Exchange server, that can send e-mail (I will explain this one below) sends mail directly to the recipient server (actually, to the server, closest to the recipient, that is capable of receiving mail, but again, I’ll explain it below) no matter how far away it is. Even if there is no direct site link between sites. In Exchange 2003 terms, all the transport-enabled servers in Exchange 2007 Organization behave as if they were one large Routing Group. You can change it by specifying “hub sites”. In this case, if there is no direct link between sites, Exchange will use servers in a “hub site” for relay.

Store or Send

In Exchange 2003 mailbox servers send and receive mail. In fact, the only servers that don’t send or receive mail are the servers that don’t have mailboxes: front-end servers, public folder servers, etc. In 2007 mailbox servers and transport (”hub”) servers are separate. You can have them on one physical machine, but those are two distinct software components. In essence the process of transporting e-mail is the following:

1. User hits Send
2. Mailbox server signals closest hub server that it has mail to send. Closest hub server is calculated based on AD sites topology.
3. Hub server establishes MAPI connection to the mailbox server and retrieves mail
4. Hub servers sends mail (using SMTP) to the hub server closest to the recipient’s mailbox server. That is if it’s a different hub server.
5. This second hub server uses MAPI to put mail int the recipient’s mailbox.

BTW, all the network traffic in the process above is encrypted. It’s either encrypted RPC or TLS SMTP. This server is the same even if you send mail to the same mailbox server or even to yourself

Using Computer Names in Connection Control for IIS and Exchange

Add comment June 9th, 2006

Let’s start with some well-known facts. There is a possibility to restrict IIS or Exchange Virtual Server (that is Website, SMTP, POP3, or IMAP4 Virtual Server) based on IP addresses and computer name. In order to use this feature, you need to display Virtual Server Properties window in IIS Administrator or Exchange System Manager, click on Access, tab and then click on Connection button. Then, you will see the following familiar window:



If you click Add in this window, the following window will appear:



Here, you can specify exceptions to the general rule selected in the first window. Up until now, I have been stating widely known facts. It’s pretty obvious how this option works with IP addresses, but the computer names is the whole other story. And that’s what I’d like to discuss.

It appears as if you can specify the domain name here and all the computers in this domain will be treated as exceptions. Well, that’s not true. First of all it’s not a domain. The name you specify here must be the exact match of a computer name.

But even after I said that, there is still a question. Which name to match? Short host/NetBIOS name or FQDN? Good question. That depends. The server performs a reverse DNS query based on the client’s IP address. If the query is successful, the result of that query is matched against the exceptions list. The result, of course, is usually FQDN. If that query fails, server uses whatever name supplied by the client which is usually the host/NetBIOS name.

I hope this helps. Oh, wait… I hope it doesn’t. Because it’s strongly recommended not to use computer names in that exceptions list. Try to use IP addresses only. Names in that list put a huge burden on a server because it needs to perform a reverse DNS query against every SMTP client.

Now I hope this helps…

Outlook and Protocol Sequences

Add comment June 9th, 2006

I was recently testing different troubleshooting scenarios for MAPI client connectivity issues.

MAPI clients connect to the Exchange IS service, and there are a few ways to do that. Normally, Exchange 2003 registers six interfaces (provided, the only installed network protocol is TCP/IP). There are only two that relevant to a MAPI client (Outlook):

1. “ncacn_ip_tcp” - uses TCP. Uses one of the higher (1024 and above) TCP ports randomly (that is by default) selected by the server when Exchange IS starts. This can be manually configured on a server - see Knowledge Base Article 270836 for details
2. “ncacn_np” - uses Named Pipes. Uses TCP port 445.

Outlook XP attempts to use ncacn_ip_tcp first. If this fails, it falls back to ncacn_np. From my experience it works much slower through ncacn_np. Outlook 2003 only uses ncacn_ip_tcp. In TCP ports term this means that if you mistakenly block higher ports, Outlook XP will work slow.

As you can see from the above, ncacn_ip_tcp is the most important protocol sequence from the Outlook connectivity standpoint. In order for this protocol sequence to work, you need to make sure that two TCP ports on the server can be reached by the client:

1. Port 135
2. Higher port selected by Exchange IS.

Port 135 is used by the RPC Endpoint Mapper, so the client needs to reach it in order to receive information about the TCP port that is actually used by Exchange IS.

There are two ways to determine (for troubleshooting purposes) the port used by the Exchange IS:

1. We can simply define it as described in Knowledge Base Article 270836
2. We can use rpcdump-like tool

Rpcdump is a part of Windows 2003 Resource Kit and available for download from the Microsoft web site. It simply gives you the list of RPC interfaces registered on the server. Run in in verbose mode (-v) and look for the interface with protocol sequence “ncacn_ip_tcp” and UUID “a4f1db00-ca47-1067-b31f-00dd010662da“. The TCP port number will be in brackets after the IP address. You can use a simple telnet client to test connectivity.