Upgrading Fujitsu Servers from ESXi 6.5 to 6.7

I had a task to update a number of Fujitsu Primergy BX2580 M1 servers from VMware ESXi 6.5 to ESX 6.7. All of them were on VMware ESXi 6.5 build 17167537. Most of them had been build with Fujitsu-VMware-Installer-6.5-10175896-V421-1 custom ISO. These upgraded without an issue to 6.7 with VMware Update Manager using the Fujitsu-VMvisor-Installer-6.7-14320388-v480-1 custom ISO. However, some of them had originally been built with the Fujitsu-VMvisor-Installer-6.5-13932283-v430-1 custom ISO, as shown in the screen shot below


When checking the compliance of these hosts using the HTML-5 vSphere Client the scan task failed and they showed with an unknown baseline, as below.


The scan task reported a “Cannot execute upgrade script on host.” error, as below.

I also got this error if I tried to upgrade the host by remediating it.

On checking the /var/log/vau.log on the host I discovered this error “Cannot merge VIBs EMULEX_bootbank_emulex-esx-brcmfcoecli_12.0.1113.0-01, EMULEX_bootbank_emulex-esx-brcmfcoecli_12.0.1113.0-01 with unequal payloads

Stripping off EMULEX_bootbank from the start of the VIB name and the version number at the end, I could confirm that the emulex-esx-brcmfcoecli VIB was installed on the host with command:

    esxcli software vib list | grep emulex-esx-brcmfcoecli

As this VIB was not being used, I removed it with:

    esxcli software vib remove –vibname=emulex-esx-brcmfcoecli

I then ran a scan (Check Compliance) on the host again, and once again it failed with the “Cannot execute upgrade script on host.” error again. Checking /var/log/vau.log reported a problem with another VIB.

I removed this VIB and performed another scan. I had to repeat this a number of times, each time it reported a single VIB that it could not “merge”. Once I had removed all the VIBs it had an issue with the scan worked and the upgrade baseline was listed as non-compliant on the host. The complete list of VIBs I had to remove were: –

  • emulex-esx-elxmgmt
  • elx-esx-libelxima.so
  • emulex-esx-elxlpfccli
  • emulex-esx-brcmfcoecli
  • emulex-esx-elxnetcli

Removal of elx-esx-libelxima.so required the hosts to be rebooted.

As I had a quite a few of these to do but they were in clusters where I could take half a dozen off line at the same time I remove these VIBs using PowerCLI and running a different Powershell session for each hosts so I could do them concurrently as it took a while for the VIBs to be removed. The code I used was:

    $esxcli = Get-esxcli -V2 -VMHost <hostname>

    $VIBS = $esxcli.software.vib.remove.CreateArgs()

    $VIBS.vibname = “emulex-esx-elxmgmt”, “emulex-esx-elxlpfccli”, “emulex-esx-brcmfcoecli”, “emulex-esx-elxnetcli”, “elx-esx-libelxima.so”


Posted in Uncategorized | Leave a comment

vROps – Virtual Machine CPU Stress

Now that I have vRealize Operations Manager (vROps) working again I have discovered that out of the 572 active alerts 359 of them are for “Virtual machine has continuous high CPU usage causing stress”.

I have taken a closer look at these and noticed that one of them was a virtual machine that had been provisioned with SQL Server installed on it but wasn’t being used and there were no user databases on it.

By default, vROps considers a virtual machine to have CPU stress if the CPU demand goes over 70% of the configured CPU. This virtual machine has 2 vCPUs and is running on a host with 2.6Ghz processors in it, so vROps calculates the virtual machine has 5.2Ghz CPU capacity. Therefore, if the CPU demand goes over 3.64Ghz then it will be considered to be stressed.

With the standard vROps policy this alert looks at the worst 1 hour period over the last 30 days. Looking at the CPU demand on this virtual machine over the previous week I could see that between 6pm and 8pm it was using 100% of the CPU most of the time. In fact every Sunday for a period of about 2 hours this virtual machine uses very high CPU, the previous Sunday it was 10am until 12pm and the week before 1pm to 3pm. Further investigation showed that all of the virtual machines had high CPU at some point on a Sunday due to virus scanning. I don’t really care that the virtual machines had high CPU on a Sunday, so I changed the vROps policy to only analyse data for virtual machines Monday to Sunday.

This has reduced the amount of alerts for CPU stress by a considerable amount but I am still getting some due to some of the virtual machines using more than 70% of CPU for a period of 1 hour at some point over the previous 30 days. To reduce the amount of alerts I am receiving for CPU stress further I have configured the policy to consider the entire 30 day range instead of a sliding 60 minute window.

Posted in VMware, vROps | Leave a comment

vRealize Operations Manager

I thought it was time to have another look at vRealize Operations Manager (vROps). The last time I used it was probably version 5.7 and called vCenter Operations Manager (vCOps). I have access to a couple of vSphere 6 environments with vROps 6.5 installed:

  • A production environment spread across two sites with a vCenter server at both sites, Site Recovery Manager, some VMware vSphere Metro Storage Clusters, approximately 60 ESXi hosts and 800 virtual machines
  • A Test and Development environment on a single site with a single vCenter server, approximately 35 ESXi hosts and 600 virtual machines

The 1st thing I notice is that the initial screen when you login is completely different to the older version I was used to using.

In vCOps 5.7 it looked something like this

With vROps 6.5 it looks more like this

It took me a while to get used to this new layout, but I soon preferred it. I like to go into vROps, from the home dashboard along the top I can select vCenter Server, Cluster Compute Resource, Host System, Virtual Machine and Datastore one at a time and quickly look at any Health, Risk and Efficiency alerts.

When I first got access to these environments they had not been looked at for a while and vROps was not working in either of them. In the Production environment the issue was with disk space full on the vROps appliance and in the Test and Development environment the DNS server had been changed but vROps had not been updated and therefore it could not resolve any of the host names. Both were fairly easy fixes.

I will dig deeper into vROps and post additional details along my journey.

Posted in Uncategorized | Leave a comment


I had a requirement to copy files to some web server VMs running in a DMZ that I could not access across the network. The only connectivity I had to them was via the Virtual Machine Console and logging on as a local user. The files to be copied were from an internal application server, I was adding a vDisk to this application server, disconnecting it and then adding the vDisk to the web servers (as there was a datastore shared between the hosts the application serer and web servers were running on).

I then discovered the PowerCLI command Copy-VMGuestFile. Using the following command I could copy the file I required from the application server to my local workstation.

Copy-VMGuestFile -Source E:\MyApp\WarFile\MyAPP.war -Destination D:\Temp\MyApp.wap  -GuestToLocal -VM MYAPPServer01

I could then copy the file up to the Web Servers with the following command

Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM MYAPPWeb01 -GuestUser Administrator -GuestPassword xxxxxxxxxx

Note on the command to copy the file FROM the application server I use the parameter -GuestToLocal and on the command to copy the file TO the web server I use -LocalToGuest. Also because the user I am logged on to my local workstation as does not have any permissions on the web server I have to specify a user account and password that does have permissions on the web server, I used the local administrator account, with the parameters -GuestUser Administrator -GuestPassword xxxxxxxxxx, I didn‘t need to specify an account on the command to copy from the application server because the account I was running the PowerCLI session under had permissions to this server.

If the file already exists in the destination you need to include the -Force parameter, e.g.

Copy-VMGuestFile -Source E:\MyApp\WarFile\MyAPP.war -Destination D:\Temp\MyApp.war -GuestToLocal -VM MYAPPServer01 -Force
Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM MYAPPWeb01 -GuestUser Administrator -GuestPassword xxxxxxxxxx -Force

Instead of listing the Guest Username and Password on the command line you could prompt for the credentials instead, e.g.

Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM MYAPPWeb01 -GuestCredential (Get-Credential)

As I had a number of web servers to copy the file to and they were all named MYAPPWeb with two digits at the end, e.g. MYAPPWeb01, MYAPPWeb02, e.t.c. I used the following to copy the file to all of them

$webcred = Get-Credential -Message “Please provide credentials for the Web Servers”
Copy-VMGuestFile -Source E:\MyApp\WarFile\MyAPP.war -Destination D:\Temp\MyApp.war -GuestToLocal -VM MYAPPServer01 -Force
ForEach($WebServer in Get-VM MYAPPWeb??) {
Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM $WebServer -GuestCredential $webcred -Force

Posted in Uncategorized | Leave a comment

VMware Cloud on AWS Now Available in EU West (London) Region

On the 7th March 2018 VMware released SDDC Version 1.3 of their VMware Cloud on AWS service which added the AWS EU West (London) Region. This expands on the original US West (Oregon) and US East (North Virginia). US West (Oregon) was the 1st region VMware Cloud on AWS was available in from September 2017 with US East (North Virginia) added on 28th Nov 2017 as part of SDDC Version 1.2. Addition European and Asia Pacific Regions will be added in the future. These will probably be Frankfurt soon and then Asia Pacific in the second half of 2018.

This may be a benefit for companies that have restrictions on their data residing with the UK or Europe and also allows European companies to locate their services on VMware Cloud on AWS closer to them.

Posted in AWS, VMware, VMware Cloud on AWS | Leave a comment

VMware Tools Upgrade

Recently I had to upgrade VMware Tools on a number of VMs due to ESXi being upgraded on the hosts. I remember a few years ago that there was talk of being able to upgrade VMware Tools without having to restart the VM. Great I thought but I couldn’t remember which version this came in with but as my VMs were on a version of VMware Tools that came with ESXi 6.something and I was going to a later version of 6.something then I was sure that I was going to be able to upgrade VMware Tools on my production VMs without having to restart them.

After a bit of research I discovered that yes you could upgrade VMware Tools without rebooting the VM but any driver updates required a reboot on Windows servers and running the VM for a prolonged time without rebooting following the VMware Tools upgrade was not recommended.

If you just right click on a VM and select “Upgrade VMware Tools” it WILL reboot the VM without warning you, although I think you can put an advanced option in to suppress the reboot, it is something like /reallynoreboot or you can perform a VMware Tools Upgrade from PowerCLI without a reboot e.g.

 Get-VM <vmname> | Update-Tools -NoReboot

You can use what you like to select multiple VMs at once, e.g.

 Get-VM WINSRV* | Update-Tools -NoReboot

(to get all of the VMs starting WINSRV)


 Get-VM –Location Production | Update-Tools -NoReboot

(to get all of the VMs within the Production folder – think this includes subfolder)


Get-VM –Location ESXCluster01 | Update-Tools -NoReboot

(to get all the VMs on the vSphere ESXCluster01 cluster)

If you are selecting multiple VMs then it is probably best to also add the -RunAsync option on to the Update-Tools command so that they are updated at the same time and not sequentially, e.g.

Get-VM –Location ESXCluster01 | Update-Tools –NoReboot -RunAsync

You can also update the Tools using VMware Update Manager, there is a Predefined baseline for VMware Tools Upgrade to Match Host. You attached this baseline to a VM folder, cluster or even the top level, i.e. the vCenter Server. When you remediate this baseline you have the option of running it immediately, scheduling it at a specific time, e.g. 7am, or at the next Power Cycle. Of course, even if you have attached the baseline to the top level you can select to remediate at a lower level (even down to a single VM). It would be worth attaching this baseline to the top level if only to use it to scan for VMs that are non-compliant. There is also a baseline for upgrading the VM Hardware and can be used to confirm all the VMs were on the latest VM Hardware.

On some of my VMs I used VMware Update Manager to update VMware tools at next power cycle. When you do this it enables the option in the VM to “Check and upgrade VMware Tools before each power on”, see below

I rebooted one of the VMs to confirm that it would work. It does not update tools before the reboot but after the VM powers back on, so once booted up VMware Tools are updated then the following message is displayed on the console (as you are unlikely to be on the console you will not see the message) and the server reboots a 2nd time.

It appears to leave the “Check and upgrade VMware Tools before each power on” option set on the VM, so next time there is a VM Tools update it will update the VMs again at the next power on of the VM. Not sure if I like the fact this is left on or not. Also, you need to manually do a scan of the baseline against the VMs again for the updated VMs to change from non-compliant to compliant.

When you schedule the VMware Tools upgrade with VUM you can select an option to create a snapshot before upgrading. The good thing about this is that you can set how long the snapshot stays for before it is deleted, the default is 36 hours (maybe this can be changed in the VUM settings) but I reduced it to 12 hours and in one case only 2 hours. All of the snapshots were removed automatically.

I had a problem with using VUM to update some of the servers that I scheduled to run and set them to create a snapshot. Included in the list were a couple of the clustered SQL servers. I could see that the task failed in recent tasks but could not find an error message to explain why but assuming it was down to not being able to snapshot the clustered servers so I ran it again on these two servers WITHOUT a snapshot and it worked.

Using the option to update at next reboot is a risk as it might catch someone out. Also doing a VMware Tools upgrade and suppressing the reboot causes a risk as if there is a problem with one of the new drivers it will not get detected until the server is rebooted by which time you probably don’t remember that VMware Tools was updated so makes troubleshooting the issue harder.

Posted in VMware, VMware Tools, vSphere | 1 Comment

VMware on AWS Update

I had a session with Adam Osterhost, @osterholta, from the VMware Cloud SET Team last week to update vExperts on the soon to be released VMware on AWS Service.

This service looks awesome, here are some notes I made from the session.

Clouds are becoming the new Silos, e.g. we are starting to see different clouds such as VMware on-premise private clouds, Microsoft Azure, Amazon Web Services, and Google Cloud Platform all being treated as Silos as they have their own management tools and skills required to look after them.

VMware are working on a Cross Cloud Architecture that will sit above these different clouds to provide a single architecture providing a common set of tools for: –

  • Management and Operations
  • Network and Security
  • Data Management and Governance

They already have VMware Cloud Foundation that sits above the following: –

  • On-Premise vSphere
  • VMware vCloud Air
  • vSphere running on the IBM Cloud

and soon vSphere running on AWS

This will be expanded to allow management of cloud specific non-VMware based infrastructure such as: –

  • AWS
  • Google Cloud Platform
  • Microsoft Azure
  • IBM Cloud

Use cases for VMware on AWS include: –

  • Maintain and Expand
    • Regional Capacity
    • DR and Backup
  • Consolidation and Migrate
    • Data Centre Consolidation
    • Application Migration
  • Workload Flexibility
    • Dev, Test, Lab and Training
    • Cyclic Demand

VMware on AWS will bring operational consistency while leveraging existing customer skill-set and tools across their IT environment.

It will initially be running: –

  • vSphere 6.5 on bare metal AWS hardware and not nested ESXi
  • All-flash VSAN and you will have full control of it to use or switch off features such as
    • Compression
    • Deduplication
  • NSX
  • vCenter

You will be able to use vCenter Linked Mode to your on-premise vCenter.

As VMware on AWS is using standard VMware technology then anything that works with vCenter will work with VMware on AWS such as

  • 3rd party products
  • other VMware products such as vRealize Operations
  • your own developed scripts

You will be able to use all of the AWS Services with VMware on AWS such as: –

  • Amazon EC2
  • Amazon S3
  • Amazon RDS
  • Amazon IoT
  • Amazon Direct Connect
  • Amazon IAM

As VMware on AWS will be a fully VMware managed service and they will control when vSphere is upgraded then they will introduce new functionality to allow vCenter Linked Mode to work with the same version and the previous version, i.e. on day 1 you will be able to use linked mode with VMware on AWS running 6.5 with your on-premise 6.0 vSphere infrastructure. When vSphere vNext comes out and the VMware on AWS is upgraded to it then as long as you have already upgraded your on-premise infrastructure to 6.5 then the Linked Mode will still work. This gives you more control over when you upgrade your on-premise infrastructure instead of forcing you to upgrade your on-premise when VMware on AWS is upgraded.

For best functionality, it is recommended that you have your on-premise infrastructure running the same version as VMware on AWS so that you can utilse the new functionality in the later version.

For VMware on AWS, as new VMware features only need to be tested on one type of hardware then this will allow VMware to introduce new functionality into the VMware on AWS Service before it is generally available for on-premise SDDC.

VMware are currently working with a few customers to develop the VMware on AWS Service and there will be a beta program opening up in the Spring with General Availability planned for Mid 2017

The service will go live in Virginia first and then EMEA, probably UK or Ireland, but will expand to all AWS regions/zones.

With the initial offering all hosts will have to be in the same Amazon Zone but at a later date you may be able to split across AZs for availability.

The VMware on AWS service will be sold on the number of hosts you require, not on the number of VMs you want to run. This will allow you to oversubscribe the hosts if you wish to run more VMs. The minimum number of hosts that can be purchased is likely to be 4 and will scale up. There was mention of up to 16 hosts on the session with Adam but the final details have not been worked out yet so it may be even more than that, he did show screenshots displaying options for 64 hosts.

The cost of the service will work out less the longer you commit, e.g. if you purchase a 2 year service then the cost per month will be less than if you only commit to the service for a single month.

This will be a VMware Service, delivered, operated, sold and supported by VMware. If you need support you contact the same VMware Global Support that you contact when you need support with you on-premise VMware infrastructure. You never need to contact Amazon, if there is an issue with the AWS side of things then VMware will deal directly with Amazon and therefore you are never passed backwards and forwards between the two with them pointing the finger at each other.

There will be a new web portal for requesting the service which will have a full RESTful API from Day 1 allowing features such as provisioning, scaling and billing. This web portal will be based on HTML5 for the best performance and good response times. You will be able to use the Flex Client but 90% of the functionality will be available from the HTML5 client and this should be enough for most people.

You will be able to right click on your AWS vSphere Cloud Cluster and select resize, select how many hosts and within minutes, yes minutes, you get the new capacity, i.e. additional hosts. Imagine doing this with your on-premise cloud. How long does it take to expand a vSphere Cluster on premise from the day you decide it needs to be expanded? You need to get a specification for the new hosts, qet a quote from your hardware provided, order the hardware, install the hardware, configure the hardware, install and configure ESXi, e.t.c. it can take weeks to do this. Remember the days when you had to do this every time you needed a new server and now you just provision a new VM in minutes. With VMware on AWS adding additional capacity will be like adding new VMs.

You will be able to utilise Elastic DRS which can resize your VMware on AWS Cloud by adding or removing hosts depending on workload resource requirements.

There will NOT be a dedicated Management Cluster so vCenter, NSX Manager, e.t.c. will be running on the same hosts as the VMs you want to run in this cloud but in different resource pools.

If you do not currently have NSX then you will be able to deploy a no-charge NSX Edge appliance within your on-premise infrastructure to allow functionality with VMware on AWS.

When VMware upgrade the VMware on AWS service they will add an extra host into your cluster so that one host can be taken out of service to be upgraded so you do not lose any capacity.

The specification of each host will be very similar to the Amazon I3 specification.

You will be able to vMotion between your on-premise infrastructure and VMware on AWS.

For more updates on VMware on AWS you can follow @vmwarecloud on twitter or search for the hashtag #VMWonAWS

Posted in AWS, VMware, VMware Cloud on AWS | Leave a comment

VMware on AWS

Coming soon VMware on AWS.

Soon you will be able to run VMware’s Software Defined Data Center (SDDC) including vSphere, vSAN and NSX on the Amazon Web Services (AWS) Cloud. It is currently in Technology Preview but expect more details regarding general availability soon.

This will mean that you will be able to run any application across vSphere based on premise private cloud, public and hybrid clouds environments.

It will be sold and supported by VMware as an on-demand service utilising Amazon’s global enterprise-grade secure highly-scalable cloud resources.

Some of the customer scenarios this is targeted at are: –

  • Application Development
  • Testing
  • Disaster Recovery
  • Geographical Expansion
  • Burst Capacity
  • Data Centre Migration

You will be able to manage it from an existing vCenter Management Interface.

By utilising VMware on AWS should allow companies to free up time spent managing infrastructure to spend more time innovating.

Dr. Matt Wood, AWS GM – Product Strategy, and Mark Lohmeyer, VMware VP Products, demonstrate how a VMware SDDC can be spun up in a few clicks and a matter of minutes in the video below. It is simply a case of selection a geographic location for the Data Center you want to spin up, the size of the Data Centre and how you want to pay for it. They also demonstrate how you can vMotion a Virtual Machine from an on-premise private VMware cloud to VMware on AWS just as you would vMotion on VM between hosts within your own private vSphere environment with no downtime.


It looks like this will also include Elastic DRC, where a new host will be spun up on AWS when there is a shortage of resources to satisfy all of the active workloads and DRS will rebalance the workloads across the infrastructure including the new host(s).

For more information see https://www.vmware.com/cloud-services/vmware-cloud-aws.html

Posted in AWS, VMware, VMware Cloud on AWS | Leave a comment

VMware Snapshot Performance Impact

A number of people have asked me recently about the performance impact of having VMware snaphots on a VM. So here is my understanding of the performance impact of having snapshot(s) on a VM.

The issue with having a snapshot on a VM is that when a block of data is changed instead of changing this in the main vmdk file it is written to a delta file.

The delta files are not pre-allocated as the main vmdk file is (assuming it is thick provisioned); so each time you have to write to the delta file first off the delta file has to be extended and then the data written. Well not exactly every time data is written, I think it extends the delta files in 16MB chunks, so only has to extend for every 16MB of new blocks that have changed. If you change the same block of data multiple times after a snapshot is created I believe it overwrites the block in the delta file so does not need to extend the file for blocks that are changing multiple times. When the files are extended it locks the VMFS volume (or it used to, not sure if this is still the case with vSphere 6.x) so it is not just the VM with the snapshot on it being impacted but also any other VM on the same datastore.

This is the reason a thick provisioned vmdk provides slightly better performance than a thin provisioned vmdk; as the space is pre-allocated in a thick provisioned vmdk therefore the file does not need to be extended to write additional data to it. If performance of the vmdk is really important then you should create eager zeroed thick provisioned disks as all blocks are zeroed in the vmdk at the time of creation, with the default of lazy zeroed thick provisioned disks each time a block is written to for the first time it needs to be zeroed before the data is written to it; don’t ask me why it needs to be zeroed and then overwritten as I don’t have a clue! Just know that eager zeroed is supposed to be better but takes longer to create the vmdk in the first place.

When you want to read a block of data from a VM with snapshot(s) on it the vmkernel needs to work out if the block you want to read is in the delta file or in the base vmdk.

I have heard that the more snapshots you have on a VM the bigger the impact on performance and have seen a VMware KB (https://kb.vmware.com/kb/1025279) suggesting that you should not have more than 32 and for better performance only 2 or 3.

This extra processing has an impact on the VM disk access times, I suspect not a great lot of impact but it is not as quick as going directly to the base vmdk to read and write blocks of data. I think you will only ever notice it, if at all, with very high disk I/O intensive workloads.

When I get change I will build a test environment and run some performance tests to see how much of an impact snapshots have, although it will probably depend on your disk subsystem.

Posted in VMware, vSphere | Leave a comment

VMware vSphere 6.5

VMware vSphere 6.5 was released earlier this week, it was announced on 18th Oct 2016 at VMworld Barcelona but was officially available for download from Tuesday this week (15th Nov 2016).

Some of the new features I got to look at as part of the beta program and thought were good additions to the product were: –

  • Platform Services Controller (PSC) HA – in vCenter 6.0 we could implement HA for the PSC but this required complex configuration and the use of a load balancer, with 6.5 no load balancer is required and the configuration is much simpler.
  • vCenter HA – in the past we have had things such as VMware vCenter Heatbeat to provide HA of the vCenter but that was withdrawn some time again. Now we have an easy to setup vCenter HA solution.
  • VMware Update Manager (VUM) now included in the vCenter Server Appliance so you no longer require a Windows Server for VUM.
  • vSphere HA Orchestrated Restarts – you can now configure dependencies between VMs, e.g. an App server will not restart until a SQL server it is dependent on has been restarted.
  • Additional HA Restart Priorities – up to vSphere 6.0 you could configure one of 3 restart priorities for a VM, other than disabled, (high, medium, low). Now you have 2 more (highest and lowest) making a total of 5 – similar to Site Recovery Manager (SRM).

There are loads more new features but these are a few that I wanted to pick out and will dig unto them in greater detail in later articles.

Posted in VMware, vSphere | Leave a comment