Mar 022010

Since configuring Multicast NLB in ISA Server Integrated Mode is quite a lot of work, I decided to write something down about it in this blog. First of all, there are some requirements before starting. So first let’s gather all the data, tools and etc. before continuing.

  • It’s a requirement to be able to create static ARP entries on the router or routing device.
  • If you already have NLB configured in your environment then document your current NLB configuration carefully (for example: IP addresses) before switching to multicast.
  • Make sure that ISA Server 2006 is at least on SP1.
  • Contact Microsoft to request the script from the Microsoft KB or download it directly from here.
  • Download NLBclear from the Microsoft website.
  • Also download this hot fix that disables the SNP (Scalable Network Pack) features written about in this KB.

Well ok, now that we’ve all followed and downloaded the requirements, let’s start working on this. I already assume that you are just like me and keep your ISA 2006 server fully patched so I’m not going to explain how to install SP1 for ISA. If you didn’t installed it yet, do it right now and follow the “next, next, finish” wizard. Keep in mind that you also update your CSS server if you have a distributed environment.

Ok, first of all we need to document our current configuration. Personally, I’m using the ipconfig /all command on every node to make sure I have all the IP addresses per interface. I actually use two commands. You can run those commands at any time, even right now. 😉

    1. C:\>ipconfig /all > C:\IpNode1.txt
    2. C:\>wlbs display > C:\WlbsNode1.txt

    With both commands I’m sure I have all the data collected about the current NLB configuration. Of course it’s always a good thing to backup your configurations so I assume you already have a regular backup strategy in place in case something goes wrong.

    So now we have collected our data and we can start installing the SNP fix. SNP can give you nasty problems with your cluster or with ISA in any way. Symptoms include loosing connectivity or any other IP based problems. Anyhow, it’s all written in the KB I already directed you to above. Again, it’s just a “next, next, finish” wizard so there’s nothing to explain about it. 🙂

    Removing the NLB configuration

    If you haven’t configured NLB on your ISA servers you can skip this part. If you have, then keep reading. 😉

    Ok, now ISA server is completely ready to be re-configured to a Multicast NLB configuration. But first, we need to disable NLB within the ISA console. To do this, 1)Open up the ISA server console, 2)In the left pane expand your array, 3)Go to the configuration option and choose Networks, 4)In the middle pane choose the Networks tab again if it is not selected, 5)In the right pane (the Swoosh bar) you will see the screen in the image below. Click on the Disable Network Load Balancing Integration.

    Disable NLB Integrated

    You will receive a warning as you can see below. If you click “Ok” and “Apply” the ISA server clears it NLB configuration… right?Warning clearing NLB Well, No. Not completely. It does remove the data out of the ISA server configuration, but there are still some leftovers on the systems themselves. For example, If you check it out you will notice that you still have multiple IP addresses configured on the IP properties of your network interface cards. It’s bad that ISA doesn’t remove this, but the good news is that TMG does remove it.  Also, TMG is much easier to configure for Multicast NLB, as I wrote earlier.

    Anyhow, the next step to completely removing the NLB configuration on the ISA servers is to use the NLBclear utility that you downloaded earlier. The NLBclear command should be run on each node within this array. I know, loads of work… well, not really.  NLBClear.exe is not actually the utility that you use. In fact it’s just extracting the correct files you need. When you extract NLBclear.exe you will find a couple of files. However, just one file is the one to execute and that is RemoveAllNLBSettings.cmd. Just open a command prompt, locate the folder and execute the batch file. If you do this right you will see a screen similar to this:

    Example NLB clear 
    It is merely asking you for your permission and also warning you that the Firewall service will be stopped. We need to do this since we are planning to reconfigure the NLB settings completely. This script will definitely remove all the previous configured settings you made earlier. It’s also a great tool to use if you have any problems and want to reconfigure NLB. 😉

    Reconfiguring NLB

    Good, now it’s time to reconfigure the NLB configuration to support multicast. With the tool previously provided you have multiple options, including reversing back to unicast NLB which is the default with ISA server. There is a readme file included in the file, however I think it’s a good idea to copy and paste it in here. At least the examples:

    — To operate on the current array:
      cscript KB938550.wsf /array:. /nlb:unicast /net1:netname

    — To operate on a specified array:
       cscript KB938550.wsf /array:arrayname /nlb:unicast /net1:netname

    — To define NLB Unicast for a single network:
       cscript KB938550.wsf /array:arrayname /nlb:unicast /net1:netname

    — To define NLB Multicast for a single network:
       cscript KB938550.wsf /array:arrayname /nlb:multicast /net1:netname

    — To define NLB Multicast with IGMP support for a single network:
       cscript KB938550.wsf /array:arrayname /nlb:igmp /net1:netname

    — To operate on multiple networks:
    cscript KB938550.wsf /array:arrayname /nlb:multicast /net1:netname /net2:netname /net3:net
       – There is no limit to the number of networks that can be managed in a single script execution

    Most of the readme file speaks for itself. But please note that you have to run this from the command prompt. Also the ‘netname’ is in fact the names configured in the networking tab within the ISA console. Not the Network adaptors or anything like that. For example see the screenshot:

    Example configuring NLB IGMP

    Before continuing, please check to see if the CSS servers are fully replicated. You can check this out in the monitoring part of the ISA console.

    Ok, we’re almost done. In fact, there is only one more thing to do within the ISA console and that is re-enabling the NLB configuration. To do this, open up the ISA server console again, expand your array from within the left pane and then go to the configuration option and choose Networks. In the middle pane choose the Networks tab if it isn’t already selected.  Look In the right pane (the Swoosh bar). This all sounds familiar so far, huh? Well indeed, but now instead of Disable Network Load Balancing Integration it has changed to Enable. 😉
    Enable NLB settings

    Well, just 2 more things to do. First of all, after enabling NLB we should also configure the NLB networks correctly. As soon as you hit the Enable button you will get a wizard where you can configure the NLB configuration.  You will see a screen similar to this image:

     Configure NLB settings

    Final steps

    Ok, now we are almost set. Finally. The last thing we should do is to set static ARP entries in the routing device near ISA. Usually this can be done on a core switch or a switch near the ISA server. But to do this you first need to find out the current configured Multicast MAC addresses. To find this quick and easy, you cab simply use the WLBS ip2mac command line tool. With this command you will get an output similar to this:

    wlbs ip2mac example

    So, simply enter the command “wlbs ip2mac <network name found in ISA>” and you will find all the MAC addresses you might need.

    So now you found the the MAC addresses used by the ISA server cluster and you can finally configure your switches and/or routers. To do this check out your switch/router vendor’s documentation. For Cisco please review this article. It will explain how to configure it on Cisco Catalyst switches along with some background information.

    Feb 232010

    Ok, I know the software might be a bit old and already near the end of support but I also know it’s still used a lot. Many clients might consider not to upgrade since it’s still working. Although I don’t agree with this. 🙂

    Anyhow, a few days ago I heard there were problems with the MTU sizes from the ISA environment. In this network there are many devices behind the ISA servers and each of them could cause such issues. Although I don’t suspect it’s the ISA server yet (until I see the traces) I always looking a bit further if we are still able to resolve it on a Microsoft system, even if they are incorrect by blaming Microsoft. It’s easy to do, so why not huh? 😀 So, while searching I found this Microsoft article.

    This behavior may occur because of the TCP/IP maximum transmission unit (MTU) setting that is applied during ISA Server installation. To prevent an attacker from changing the MTU value, ISA Server 2004 disables path MTU (PMTU) Discovery. This setting is documented in Microsoft security bulletin MS05-019
    By default, Windows uses an MTU setting of 1,480 bytes and accepts Internet Control Message Protocol (ICMP) messages that request smaller packet sizes.
    If MTU discovery is disabled on a Windows-based server, the server uses an MTU setting of 576 bytes.

    However, the other interesting Microsoft article to read is about harden the TCP/IP stack. So Microsoft states: Microsoft recommends that you set EnablePMTUDiscovery to 0. When you do so, an MTU of 576 bytes is used for all connections that are not hosts on the local subnet.

    So I’m curious why Microsoft would set the MTU size to such a low value. In my opinion it makes no sense. If Microsoft would set it to a MTU value of about 1420 it would make in much more sense in my opinion. With such a low value of 576 bytes ISA have to fragment each packet and this can slows down your performance very rapidly. In environments where ISA is not directly connected to the edge this might slows down the complete network behind it. In fact, there are devices (including ISA) which can drop fragmented packets.

    Hmmm ok, but what the hell is PMTU then? Well PMTU stands for Path MTU or Path Maximum Transmission Unit. PMTU is used to determine the MTU size on the network path between different hosts usually with the goal of avoiding IP fragmentation. The server will send out packets with the Don’t Fragment bit enabled. If any device in the path who’s MTU is smaller then the packet might drop the packet and will send back an ICMP (Type 3, code 4) “Fragmentation needed” message back which contains it’s MTU size. This will allow the ISA machine to reduce his Path MTU size. This will be repeated until the MTU size is small enough to transverse the entire path without fragmentation.

    So what can we do about  it…

    • First of all I understand that Microsoft thinks about possible security issues but personally I would change the EnablePMTUDiscovery to a value of 1. If you enable it, you might also enable black hole detection which is documented in this article. When Black hole detection is enabled the MTU size will be reduced only for the problematic networks.
    • If you don’t want to enable the EnablePMTUDiscovery or it can’t be done or you have some other kind of reason why you don’t want to enable this setting back again, you might override this setting by fixing the MTU size per network adapter. Also Microsoft stated that it could be a potential security risk which you can read about it in the previous articles about the TCP/IP hardening. Changing this setting is well documented in this Microsoft article but the performance might be lower then when you would enable the EnablePMTUDiscovery key. 
    Feb 022010

    Due to security reasons WPAD and ISATAP have been by default blocked in Windows server 2008 DNS. There is a GlobalQueryBlockList which blocks these entries. When you try to create a WPAD entry to configure your ISA Auto Discovery, it will fail to resolve the WPAD entry whereas it will resolve the WSPAD entry. This can be quite problematic when you using the WPAD DNS entry rather then the DHCP possibility.

    To remove the WPAD from the block list type the following command:
    Dnscmd /config /globalqueryblocklistisatap

    This command will override the existing list with ISATAP as the only keyword.
    Now, you can resolve the WPAD entry from NSLOOKUP

    Check DNS Server Global Query Block List from the Microsoft website for more details about DNS