• Prod-Cluster
• Dev-Cluster
• UAT-Cluster

Some LUNs are mapped to more than one cluster.

Using PowerShell with the Data ONTap module installed I could get a list of LUNs with

Get-NaLun

I could then filter this to just the ones that are mapped with

Get-NaLun | Where {$_.Mapped}

I collected the results of the above command into an array named $MappedLUNs, i..e

$MappedLUNs = Get-NaLun | Where {$_.Mapped}

I then setup an object with the property names I wanted to report on, by using a trick I learned from Hal Rottenberg by piping “” to the select commandlet, i.e.

$LUNMap = “” | Select LUN, Prod-Cluster, Test-Cluster, UAT-Cluster

Then for each of the objects in the $MappedLUNs array I got the LUN path and then using Get-NaLUNMap I obtained the LUNID property where the Initiator Group Name equalled each of the Igroups I was interested in, i.e.

ForEach ($LUN in $MappedLUNs) {
$LUNMap.LUN = $lun.path
$LUNMap.Prod-Cluster = (Get-NaLUNMap $LUN | Where {$_.InitiatorGroupName -eq “Prod-Cluster”}).LUNID
$LUNMap.Test-Cluster = (Get-NaLUNMap $LUN | Where {$_.InitiatorGroupName -eq “Test-Cluster}).LUNID
$LUNMap.UAT-Cluster = (Get-NaLUNMap $LUN | Where {$_.InitiatorGroupName -eq “UAT-Cluster”}).LUNID
$LUNMap
}

I saved all of these commands into a script file named Get-LUNMap.ps1 and piped the output through Export-CSV to get the results into a CSV file which I could load up into Microsoft Excel to perform some formatting, sorting and analysis, i.e.

Get-LUNMap.ps1 | Export-CSV lunmap.csv

This is a time consuming process, especially if you need to perform it for a large number of virtual machines. Also I do not like editing the settings of a virtual machines just to look at the setting as I may accidently change something and forget to click on cancel instead of OK. So I looked at getting this information with PowerCLI.

You can get a list of hard disks the virtual machine has with the following one liner

Get-VM | Get-HardDisk

You can then limit this information to just the Raw Disk Mappings as follows:

Get-VM | Get-HardDisk | Where {$_.DiskType –eq “RawPhysical”}

However the properties returned do not include the LUN id. From the SCSI Canonical Name you can find out the Runtime Name by using the Get-SCSILun command against one of the hosts, of course this should be the host the virtual machine is running on which is returned in the VMHost property from Get-VM. So if I run Get-SCSILun for the SCSI Canonical Name of each of the Raw Disk Mappings I can get the Runtime Name. Then if I use the Substring method on the Runtime Name to strip off the number of L at the end of the Runtime Name, by using the LastIndexof method, I can get the LUNid. The following commands achieve this:

$Disks = Get-VM | Get-HardDisk | Where {$_.DiskType -eq “RawPhysical”}
Foreach ($Disk in $Disks) {
$Lun = Get-SCSILun $Disk.SCSICanonicalName -VMHost (Get-VM ).VMHost
$Lun.RuntimeName.Substring($Lun.RuntimeName.LastIndexof(“L”)+1)
}

Where is the name of the virtual machine you want to get the LUN ids for.

I have put all of this into a script and only run the commands if the virtual machine passed to the script exists, i.e.

Param($VM)

if (Get-VM $VM) {

$Disks
= Get-VM $VM | Get-HardDisk | Where {$_.DiskType -eq
“RawPhysical”}

Foreach ($Disk
in
$Disks) {

$Lun
= Get-SCSILun $Disk.SCSICanonicalName -VMHost (Get-VM $VM).VMHost

$Lun.RuntimeName.Substring($Lun.RuntimeName.LastIndexof(“L”)+1)

}

}

I saved this script in C:\VIscripts as Get-VMLUNid.ps1 and run it as follows: -

C:\VIscripts\Get-VMLUNid.ps1

e.g.

C:\VIscripts\Get-VMLUNid.ps1 sqlsvr01

Right clicking on the host and selecting Connect does not normally fix the problem. Also logging in directly to the host with the vSphere Client usually does not work, or if it does it disconnects quickly after connecting.

All of the virtual machines running on the host continue to run; however without the host being connected to vCenter or being able to connected directly with the vSphere Client you are unable to manage the virtual machines. Also, connecting to the host with PowerCLI does not work, if it does connect it drops the connection soon afterwards.

I have used a variety of “tricks” to get the host to reconnect.

1. Restart the management agents and then reconnect. If this does not work the first time then try again. I have found that it often works on the second attempt.
   1. From the host console press F2 to login
   2. Enter the root password
   3. Go down to Troubleshooting Options and select it
   4. Select Restart Management Agents
   5. Press F11 to restart the management agents
   6. Once they have been restarted attempt to reconnect the host by right clicking on it within vCenter and select Connect. You will normally get an error message and then prompted to enter a username and password, enter root and the root password.
2. If the above fails twice then try removing the host from vCenter and adding it in again. This has an impact in that you will lose performance statistics, the virtual machines will need to be put back into the correct resource pools if you are using resource pools and if you are using Site Recovery Manager (SRM) the virtual machine protection will need to be reconfigured. You might want to skip this step and try the ones below first and use this as a last resort.
   1. Right click on the host in vCenter and select Remove.
   2. Once it has been removed right click on the container the host was originally in, e.g. a cluster and select Add host
   3. Enter the host name, root for the username and the root password
   4. If the host starts to add and then fails then repeat the steps in 1 above, again you might have to try the steps in 1 above a couple of time.
3. If you still cannot get the host to reconnect and you are using Fibre Channel storage then rescan the HBAs. As you cannot manage the host with a vSphere Client you will have to do this at the command line.
   1. At the console, if you are not already in the Troubleshooting Options then follow steps a through to c in 1 above to get to the Troubleshooting Options.
   2. If the menu shows Disable Remote Tech Support Mode then Remote Tech Support is already enabled, if there is an option for Enable Remote Tech Support Mode then select it.
   3. Using a SSH client such as Putty to get a SSH connection to the host
   4. Login as root
   5. Issue the command esxcfg-rescan for each of the HBAs on the host where hba is the hba device, e.g.
      esxcfg-rescan vmhba1
      esxcfg-rescan vmhba2
   6. Now try reconnecting the host by right clicking on it and selecting Connect as described in step 1.f. Again if it does not work then follow the steps in 1 above a couple of times.
4. If you still cannot get the host to connect check for redundant directories in /var/run/vmware/root_0 and /var/lib/vmware/hostd/stats being full, tidy these directories and attempt to reconnect again.
   1. If you do not already have a SSH connection to the host follow steps a through to d in 3 above to get a SSH connection.
   2. cd /var/run/vmware/root_0
   3. There should be a directory in here for each running virtual machines on the host, issue the following command to get a list of all the directories here.
      ls

      If there are more directories than the number of running virtual machines then use the following command to remove the empty ones, it will attempt to delete the non-empty ones but will fail to delete these, so you are safe to run the command against all directories
      rmdir *

   4. Issue the following command to check for full filesystems.
      vdf

      The one to check is hostdstats, if it is 100% full then tidy it as follows

      1. cd /var/lib/vmware/hostd/stats
      2. rm hostAgentStats-*.stats
   5. Now restart all of the services with
      services.sh restart
      You can run this while there are running virtual machines on the host without affecting them.
   6. Now attempt to connect the host again, as detailed in 1.f.

I have found that if a host becomes disconnected and I have had to use the steps above to reconnect it then it becomes disconnected again within the next 24 hours or so unless the host is restarted. Therefore, I suggest that once you have the host reconnected put it into maintenance mode and restart it.

I think this issue is being caused by NetApp SnapManager for SQL and SnapManager for Exchange because usually the SnapDrive running on one of the virtual machines on the failing host is normally reconfiguring the virtual machine and rescanning the HBAs to attach or detach RDMs from a NetApp snapshot to verify a backup that has just been run by the SnapManager product running on the virtual machine when the host becomes disconnected. I do not think that it is a fault of the SnapManager product as it is just using the VMware APIs to perform the tasks it needs to do. All of the hosts I am having this issue with are running an unpatched version of ESXi 4.1 (build 260247). They are also running from IBM supplied USB keys without the latest IBM customisation. I plan to upgrade the hosts to at least ESXi 4.1 update 1 (build 45697) or ESXi 4.1 update 2 (build 502767) to see if this helps the situation. I will also apply IBM customisation 1.0.4 as this fixes the issue of vMotion and Fault Tolerance becoming disabled following a reboot of the host. I will update this post with details of whether these updates helped or not.

“Active Directory Web Services encountered an error while reading the settings for the specified Active Directory Lightweight Directory Services instance. Active Directory Web Services will retry this operation periodically. In the mean time, this instance will be ignored. Instance name: ADAM_VMwareVCMSDS”

To fix this look in the regestry key

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ADAM_VMwareVCMSDS\Parameters

for

Port SSL.

If Port SSL is already there as a string (REG_SZ), delete it and create a new REG_DWORD named Port SSL and set the value to 636 (Decimal). You then need to restart the two services “Active Directory Web Services” and “VMwareVCMSDS”. Not sure if both of these services needed to be restarted but I have always done this whenever I notice this error message without impacting anything

I have yet to discover anything not functioning while this error is occurring but it is annoying that it repeats every minute so I always ensure the registry key is correct on new implementations and correct it on any installation suffering from the issue.

vSwicth0 – vmnic0, vmnic1, vmnic2, vmnic 6 – all active
VMKernel Port – Management – enabled for Management – Active vmnic0, Standby vmnic2, unused vmnic1 and vmnic6
VMKernel Port – vMotion – enabled for vMotion – Active vmnic2, Standby vmnic0, unused vmnic1 and vmnic6
VMKernel Port – FT Logging – enabled for FT Logging – Active – vmnic1 and vmnic6, standby vmnic0 and vmnic2

Each VMKernel Port only has the function it is intended for enabled and the other two disabled, i.e. the Management VMKernel port is enabled for Management but has vMotion and FT Logging disabled.

When performing a vMotion and monitoring the physical switch I notice traffic coming out of the source host on vmnic0 and going in to the destination host on vmnic2.  This should be coming out of vmnic2 and going in to vmnic2.  It always does this regardless of the source and destination hosts, e.g. if I vMotion a VM from host01 to host02 traffic goes from vmnic0 on host01 to vmnic2 on host02, when vMotioning the VM back from host02 to host01 it goes from vmnic0 on host02 to vmnic2 on host01.

I have also tried separating vMotion and Management onto different vSwitches but still see the traffic coming out of the vmnic configured for management on the source, e.g.

vSwitch0 – vmnic0 acvtive, vnmic1 standby
VMKernel Port – Management – only enabled for Management

vSwitch1 – vmnic2 active,  vmnic6 standby
VMKernel Port – vMotion – only enabled for vMotion

When I perform a vMotion the vmnic0 is used on the source host and vmnic2 on the destination host.

This is all true when vMotion and Management are on the same VLAN/subnet, this should be a valid configuration.

If I change vMotion to use it’s own VLAN/subnet then the correct vmnic is used on the source host but I should not have to do this.

The minor issue I have with using a separate VLAN/subnet for vMotion and Management is that they both use the same VMKernel default gateway so the vMotion interface is now configured for a default gateway which it can never access!  Not a major issue as the vMotion traffic should never need to leave it’s subnet.  Some environments I have the vMotion interface is being monitored by a monitoring utility and this can not access the vMotion interfaces if the default gateway is not accessible.

The customer has two storage devices connected to the same Storage Area Network (SAN); an IBM DS4700 and an IBM DS3400. The DS4700 has Fibre Channel disks in it and the DS3400 SATA disks.

I performed a number of tests with a VM having a 22GB fat vmdk file, using the following datastores: -

VMFS_VOL2 Block Size 8MB FC Disks on the DS4700
VMFS_VOL4 Block Size 4MB FC Disks on the DS4700
VMFS_VOL12 Block Size 4MB SATA Disks on the DS3400

Details of the original tests I performed are shown in the image below, click it to see a larger version.

I then changed vmfs_vol12 to an 8MB block size to see what difference that made. Transfer times between datastores with the same block size, i.e. 8MB, where the same as the above times for transfers between datastores with matching 4MB block size. Times when the block sizes where different were double the times when the block sizes matched. Transferring to the SATA disks was slower than transferring off the SATA disks as I expected would be the case. When the block size did not match then the MB written per second where always double the MB read per second regardless of whether the transfer was from the 8MB block size to the 4MB block size or from the 4MB block size to the 8MB block size. See details below

Now I wondered what would happen if the block sizes where something other than double the other, so I deleted vmfs_vol12 and recreated it with a 1MB block size, my thoughts were that when transferring between an 8MB block size and a 1MB block size I would see the MB written being 8 times the MB read and the transfer time being 8 times the time when the block sizes matched, and when migrating between a 4MB block size and a 1MB block size the figures would be 4 times. However the MB written were once again double the MB read and almost matched the figures above for when the block sizes where different, as detailed below.

The figures in the tables above are rough figures as they fluctuated during the migration.

Here are a couple of screenshots from esxtop from when the block sizes match and when they are different.

It does not seem right that if you migrate a VM between datastores with different block sizes then the transfer takes twice as long as when the datastores have the same block size.

Further investigated required. I have posted details on the VMware ESX 4 VMTN Community, see http://communities.vmware.com/thread/293823?tstart=0

Access one of the virtual terminals, for example by pressing Alt-F1.
Login as root
Find out what virtual switch interface (vswif) the Service Console is on and what the port group is called, by defaulty vswif=vswif0 and port group=”Service Console” by issuing the command

esxcfg-vswif -l

to list the virtual switch interfaces. The output will be similar to

In the above example the vswif name is vswif0 and the port group is “Service Console”. The current IP address is 192.168.35.166, to change it issue the command

esxcfg-vswif <swif name> -p <port group> -i <new ip address> -n <network mask> -b <broadcast address>

e.g. to change it to 192.168.25.151, netmask 255.255.255.0 broadcast 192.168.25.255 enter

esxcfg-vswif vswif0 -p “Service Console” -i 192.168.25.166 -n 255.255.255.0 -b 192.168.25.255

You might also need to change the default gateway. To do this edit /etc/sysconfig/network (with something like vi) and change the GATEWAY setting, once you have edited /etc/sysconfig/network you will need to restart networking with /etc/ini.d/network restart

Extract dir6.1.1_commonagent_linux.sh from the SysDir6_1_1_Common_Agent_Linux_x86.tcdriver file (if your computer does not recognise it as a zipped file rename it to have a zip extension).
Transfer dir6.1.1_commonagent_linux.sh to the ESX host, make sure you transfer it in binary mode
Change the perfmissions on dir6.1.1_commonagent_linux.sh so that it can be executed, e.g. chmod 755 dir6.1.1_commonagent_linux.sh
If you have a Remote Supervisor Adapter installed in the server run esxcfg-module -s libsub_support=1 usb.o (you will need to restart the ESX host)
Open a number of ports on the ESX firewall with: -
     esxcfg-firewall -o 9510:9515,tcp,in,ibmdirector
     esxcfg-firewall -o 9510:9515,tcp,out,ibmdirector
Run the installer with ./dir6.1.1_commonagent_linux.sh

Once it has finished installing you should be able to discover it with the IBM Systems Director console and gain access.

Therefore the text mode had to be used.

Later when I had been given the IP addresses I set them on the service console via the vSphere Client, on checking the default gateway it was already set as it had been picked up from the DHCP server.  I checked connectivity to each of the hosts while I was oniste but later while working remotely I could not access the service consoles, obviously the default gateway was not correct. 

The servers were all IBM System x3650M2 servers with the IBM Virtual Media Key installed so I could access the Integrated Management Module (IMM) and access the console via the Remote Console functionality.  The default gatewayhas correctly listed in /etc/vmware/esx.conf but was not listed in /etc/sysconfig/network

The solution was to add the following to /etc/sysconfig/network (I added it as the last line)

GATEWAY=xxx.xxx.xxx.xxx

where xxx.xxx.xxx.xxx is the IP address of the default gateway.

For this change to be picked up networking needed to be restarted by running

/etc/init.d/network restart

This procedure could be used if you ever need to change the default gateway on ESX4.  To should be the same for previous versions of ESX.