ESXi : VM not Available (all commands greyed out)

I had a weird one today with ESXi in a standalone configuration, which means it was not connect to a vSphere host so this was a standalone host, on this host was a VM which could not powered on and none o the other functions were available they were all greyed out in the web UI

This sounds like maintenance mode is on to me and indeed it was as you can see from the host options maintenance mode is indeed on:

However what was even more weird was that the website did not allow you to make any changes, well more specially you could choose an option but it did nothing like the website could not control the host.

Local Access/iLO Access

This means we need to fall back to ESX SSH access to see what is going on however first we need to enable the SSH access so you will need local access for this which should look like this (if you see nothing on the screen press any key to wake up the display)

Then you need the F2 option, you will need to login with your valid username and password as below:

Then you need the "Troubleshooting Options" as below:

Then you need the "Enable SSH" option which will then turn to Disable SSH when enabled:

Now you have this enabled you can SSH to the host, for this I use puTTy then once you are on the device with the same username and password to access the local access we can begin with the maintenance mode issue.

Maintenance Mode Check/Remediate

First we need to check if Maintenance mode is set with this, the website UI says its enabled:

vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode

This then confirms that indeed it is enabled as you can see below:

      inMaintenanceMode = true,

How we need to disable Maintanance mode to get the server back online and allow the starting of VM's with this:

vim-cmd /hostsvc/maintenance_mode_exit

This will then cancel turn the server back to normal and stop maintenance mode, how you should be free to start the VM which will can do from the shell as well.

List VM's and Power On VM's

First we need to list all the VM's on the host which can be done with this command:

vim-cmd vmsvc/getallvms

This will then return all the VM's on the server as below:

Vmid      Name                              File                             Guest OS     Version                              Annotation

2      Honeypot   [ESX70Datastore-localstorage] Honeypot/HoneyPot1a.vmx   otherGuest64   vmx-08    HoneyPotfortheBearv2

We now need to start the VM which can be done with this command, replace the VMID with the actual number:

vim-cmd vmsvc/power.on VMID

Now we can start the VM and all should be resolved, well, no not exactly the VM failed to start because the licenses had expired, really ESX?

License Expired?

We now had an expired license, really, I think this actually failed when we lost power to the server, so the license became corrupted as you can see the license is there but the end date is blank, if you clicked on the license it reported as "never expire" with all the features enable which is wrong.

This sounds like need to get a new license to me as the corruption cannot be helping the VM being frozen, so lets generate a new license that is another trial license 

rm -r /etc/vmware/license.cfg

cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg

/etc/init.d/vpxa restart

Once you start the vpxa service, in my instance I also needed to restart the ESX host as it failed to detect the update with a simple soft reboot of vxpa.

Check license on reboot

Once the server as rebooted, go back the licensing panel and check the license and now you should see a expiry date which is more like it, a evaluation license should not las forever with all the features enabled, that would be a awful business model.

Disable SSH host on ESXi

Now we need to disable the SSH server on the host which when you click on the host options you can see the warning for, click Actions then Disable SSH - and it will be disabled, this will revert your server back to the secure defauts.