Unable to load status of objects in vCenter Server 5.1

On today’s troubleshooting, I faced a very weird problem. vCenter Server services were up and running fine, it’s able to connect by vSphere client, but VMs, hosts show gray, and I cannot power on VM via PowerCLI.

After went through each components of vCenter Server, I noticed the database size was 230GB, but only ~20 hosts were there. So I asked DBA team truncat event tables and shrink database. Issue gone after database optimization.

You may want to do same if you face similar issue.

How to change password of vCenter Server service account

Many company use service account for vCenter Server database and services. To compliance with security policy, you may need to change password of vCenter Server at regular period. This is a way I used to change password:

1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Server/Management Webservices in Services.
3. Run vpxd.exe -p command as administrator to change database password. ( It usually located on C:\Program Files\VMware\Infrastructure\VirtualCenter Server\ )
4. RDP vCenter Server by service account.
5. Open DSN of vCenter Server and click next button to save password to DSN.
6. Reboot vCenter Server.

Notes: You maybe able to logon vCenter Server if you just restart VC services, but you will face low performance to retrieve information of host, VM…etc.

Most company may has vCenter Update Manager together with vCenter Server, the password change of vCenter Update Manager service account is similar like vCenter Server.
1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Update Manager in Services.
3. Run VMwareUpdateManagerUtility.exe as administrator. ( It’s usually on C:\Program Files (x86)\VMware\Infrastructure\Update Manager\ )
4. Input new database credential on Database Setting.
5. Re-register to vCenter Server by new credential.
6. Reboot vCenter Update Manager server.

You can also reference to http://kb.vmware.com/kb/1006482 and http://kb.vmware.com/kb/1034605.

How to find which ESXi 5.1 host lock the VM

Sometimes VM may show unknown, invalid or orphan on vCenter Server, but it still running somewhere. Some technical support engineer may request reboot VM/ESXi host, or search on each host one by one.

Declare: This article only apply to ESXi 5.1, I haven’t tested on other version.

This is easiest way to find out which host lock the VM:

  1. SSH to any host on the cluster.
  2. Go to VM folder. ( Usually it’s under /vmfs/volumes/… )
  3. Run command:  vmkfstools -D “vmx file name” | grep owner
  4. Return line similar like this:
    gen 483, mode 1, owner 529495c4-0b6a7d90-a0f3-0025b541a0dc mtime 211436
  5. The red highlight section is MAC address of owner host.
  6. Run command: esxcfg-nics -l on each ESXi host to see which host match this MAC address.

Then you need to remove the invalid VM from inventory, and login to the owner host by vSphere Client and import the VMX file again.

This procedure can save lot of time to find the real owner host, but it still consumes time if it’s a large cluster. You want to more fast? It’s possible!

After you find the MAC address, change it to regular format, like: xx:xx:xx:xx:xx:xx.

Logon vMA console and connect to vCenter Server by command: vifptarget -s vCenter Server Name

Run command: esxcfg-nics -h ESXi host name -l | grep xx:xx:xx:xx:xx:xx

More fast?

Try use Excel to list commands with all ESXi host name then past on console….