Multi-path I/O (MPIO) does not work for disk with identifier … on node …

While configuring a Windows 2008 cluster, currently it has 3-nodes, getting ready to add 4th node. First step running it through cluster validation wizard.  The wizard keeps failing with the disk check returning following error:

My research indicated this might be iSCSI setting; because on Windows 2008 to use NAS technology in cluster environment iSCSI-Persist setting must be enabled.  Both [1] and [2] show how to enable that setting, but after our storage guys checked and made sure this setting was enabled we still could not get past this error. We verified flags on the HBA, the SANsurfer, and ECC Master Agent on the node with already configured nodes with no luck.

I had configured Windows 2008 Cluster (2-Node Cluster) before but never ran into this issue.  On this server we used PowerPath as agent to talk to EMC Symmetrix DMX-3.  I understanding is limited on NAS storage is limited so I didn’t know the functionality of PowerPath.  So the issue was not apparent, after looking at the 2-node cluster and the 3-node cluster we noticed the different agents were used to communicate between nodes and ECC storage.

The 3-Node cluster was configured to use SANsurfer, therefore to keep configuration on this cluster similiar we installed SANsurfer also.  However the issue remained; after reboot, we also noticed there are 59 disks visible on the server.  The server only has 2 physical disks and 29 masked; so it should be at most 30.  Looking at the cluster validation report I noticed that same disk was being masked to two ports; this is expected because of DUAL-HBA for failover.  This was same on all other nodes, so why did we not see duplicate disk on those nodes?

After digging through some documentation, talking to guys in office and internet digging, came up on [3].  Because Microsoft MPIO was not installed and not configured; each disk was mistakenly viewed as two disks.

Image from Microsoft TechNet [4]

So how to fix it now?  In Windows 2008, first you need to make sure Microsoft MPIO is installed, reference [5] on how to install it.  After you are finished installing bring up MPIO under Administrative tools.

Should see something similar to the screen above, then click on “Discover Multi-Paths“.

In this screen if you notice on the bottom you will see EMC SYMMETRIX, select that and click Add.

After which it will ask you to reboot, click “Yes”.  The problem should be resolved after reboot.

So I asked the storage expert on why did I need to configure this on this cluster and not the last one.  He said because PowerPath comes with Multi-Path I/O built in.  But because more and more people are using SAN/NAS technology with their servers Microsoft decided to add Multi-Path I/O into Windows 2008.  This makes it cheaper because then we don’t have to pay for extra license cost on PowerPath’s Multi-Path I/O.   However in his opinion, it is not as robust for load-balancing and but for failover technology (which is what I am using it for); it works without issues.

References

1. TechNet. Windows Servers. Windows Server 2008 Cluster Disk cannot bring online. Link.
2. Bryan Coffey’s Blog. Configuring Symmetrix DMX for Windows 2008 Cluster. Link.
3. TechNet. Microsoft Multipath I/O Step-By-Step Guide. Link.
4. TechNet. Microsoft Multipath I/O Step-By-Step Guide. Understanding MPIO features and components. Link.
5. TechNet. Microsoft Multipath I/O Step-By-Step Guide. Installing and Configuring MPIO. Link.
6. EMC2. EMC Symmetrix DMX-4 Series. Link.

Acknowledgment

I did lots of digging to get this working while working with co-worker, so credit isn’t mine.  I can take credit for digging, but people who came before me and storage guys are the people who had the answer to this.  I am blogging about it so 1) I have reference for future and 2) So it “might” help someone else :).