VMware, RDMs and EMC Clariion

So here’s an interesting “feature”:

I suspect most people with a Clariion also have a licence for Access Logix, aka Storage Groups, i.e. LUN masking.

If you have a Clariion but don’t have Storage Groups then all your LUNs are visible to any host that you connect/zone to the SAN, and you won’t have the issue below.

If you do use Storage Groups, then depending on how you work you may hit the problem.

So, problem number 1:

When you add a LUN into a storage group (to allow a host to see it), the Clariion gives it a new reference number that is only visbile to the hosts in that storage group. This is the HLU (Host Logical Unit), and is a simple integer, e.g. 3. The HLU is, unless you add the LUN in the right way and choose to change it, assigned automatically and is incremented each time you add a LUN to that storage group. The way to add the LUN is to go to the storage groups list,  select the storage group and click Connect LUNs, then you pick the LUN and click Add, once added to the lower pane you can click in the empty space in the Host ID column and pick the HLU you want to assign.

The LUN number you see in the main LUNs list in Unisphere is called the ALU (Array Logical Unit), it is also an integer. The host will not see the ALU, only the HLU. The HLU is the number that you see in the “LUN” column in the vSphere client.

Because the LUN Number shown in the vSphere client doesn’t tend to match the LUN number shown in the LUN list in Unisphere, I suspect most people use the NAA LUN ID to identify LUNs once they’re presented to the vSphere hosts. The NAA (Network Address Authority) looks something like: naa.6006016xxxxxxyyyyyyyyyyyyyyyyyy. The 6006016 is the manufacturer code for Data General (EMC bought them in 1999). The x’s will be a code unique to your SAN – for most of your LUNs this section will be identical. The y’s will be a LUN identifier, unique on your SAN.

When using VMFS volumes, the LUN number as seen by vSphere (the HLU) is largely irrelevant. But when you use RDMs (Raw Device Mappings – presenting a LUN directly into a VM) they become important. Especially so if for some reason you have hosts in different storage groups, and are likely to want to move a VM with RDMs between those hosts.

Consider a single LUN and two hosts, let’s say host A and host B. The LUN’s ALU is 50. The LUN’s NAA is naa.6006016048d025005af30e07fddae011. We create a Storage Group and add the LUN and both hosts into it, setting the LUN’s Host ID to 201 when we add it. Now get each host to re-scan for new storage – go to the host in vSphere client, Configuration tab, storage, Rescan All… and pick Scan for new storage devices.

When you add a new RDM, once you’ve been through the “Add…” wizard, but before you click OK on the VM Properties dialogue (aka Edit Settings dialogue), select the New Hard Disk (adding) and look in the top right of the dialogue box. There are two boxes under the title Physical LUN and Datastore Mapping File. At this point in the RDM adding process there’s some interesting information in the top box:

  • /vmfs/devices/disks/naa.6006016048d025005af30e07fddae011

Now click OK on the Properties dialogue to actually start the reconfiguration process on the VM. Once that’s completed, click Edit Settings on the VM again and once more select the new RDM and look at the Physical LUN and Datastore Mapping File boxes. Now they contain something different:

  • top box: vml.0200c900006006016048d025005af30e07fddae011565241494420
  • bottom box: [VMFS 156 FC] RCMTECHVM01/RCMTECHVM01_1.vmdk

So what’s going on here? Well you’ve just caught a glipse of how ESX actually adresses LUNs and RDMs. The /vmfs/devices/disks folder contains some interesting bits. There are two types of thing in there, use ls -l from a putty session to have a look. One is a file with the name of the NAA of every LUN presented to the host, i.e. naa.<32-digit hex>, the other is a vml.<54-digit hex> linked to an naa. file (the link is represented in the ls -l output by an arrow made from a hyphen and a greater than symbol: -> ).

That ESX does this, and how and when these files and links are created and updated is interesting as you can get into a bit of a mess if you change the HLU.

Problem number 2:

Notice how the vml link contains the NAA in the middle (starting 6006016…). Also note the 5th and 6th digits in from the left, c9 in the example above. Remember the Host ID we used when adding the LUN into the storage group, 201? Guess what 201 is in hexadecimal… it’s c9. This is a bit of a nuisance, if you change the HLU (by removing the LUN from the storage group and re-adding with a different Host ID) then you can run into issues. As per VMware KB 1016210, the procedure you must follow is to remove the LUN from the VM, then remove the LUN from the storage group, get the hosts to re-scan (as above) so that they remove the vml link to the LUN, and then re-add the LUN to the storage group with its new Host ID – now a new vml will be created containing the new HLU, then get the hosts to re-scan, then re-add the RDM to the VM. Otherwise the vml link isn’t updated with the new HLU and you can’t access the LUN.

Now let’s consider the same LUN, but this time we have two Storage Groups, host A in Storage Group HostA, host B in a storage group called HostB. Now we add the LUN to each storage group and don’t pay particular attention to the Host ID being assigned. Let’s assume that there were already a few LUNs in each storage group, also added using automatic  Host ID assignment. so the LUN is auto-assigned Host ID 4 in storage group HostA, but receives Host ID 5 in Storage Group HostB. So now consider what the vml will be for that LUN on each host: different! This will prevent you from using vMotion to move the VM between hosts in different storage groups.

Useful info:

Identifying disks when working with VMware ESX

This entry was posted in vSphere and tagged , , , , , , , , , , . Bookmark the permalink.

One Response to VMware, RDMs and EMC Clariion

  1. Kyle Barnes says:

    This is an amazing write up.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.