Tag Archives: vCenter

How to Find Out Source of Domain Accounts Locking on vCenter Server

I wrote an article talk about how to find out which services lockout domain accounts on vCenter Server. It only applies to the scenarios that domain accounts very frequently lockout. Like every 1 second. If it’s minutes, it will be hard to find out as it’s manually processing.

The other way to identify source is to use vSphere Web Client. The trick was told by VMware BCS team.

Log in to vSphere Web Client. Go to the main page and the Events node. Search for “authen” you may see some error events. The real source is red text.

User CONTOSO\test-user@ .....



在VMware Workstation上部署vCenter Server VCSA

网上有很多关于如何在VMware Workstation上部署vCenter Server VCSA的文章,但根据这些文章在部署过程中总是会遇到各种各样。以下是几点我总结出来的要点,仅供参考。

我假设你的实验环境里没有DNS或者域服务器,只是简单的使用VMware Workstation的DHCP服务,虚拟机的网卡选择的是“host-only”。以下步骤仅用于做一些快速测试时使用。

  1. vCenter Server安装好后第一次启动的时候会检测FQDN。如果你没有DNS服务器,FQDN检测会失败。所以在安装vCenter Server时要确保“Host Network Identity”输入的是IP地址。
  2. OVA文件导入后虚拟机会立刻自动启动,有时候虚拟机的网卡可能会是断开状态的。要确保网卡是连接状态。
  3. 第一次启动耗时大约15至20分钟,在没有完全启动完毕前虚拟机的控制台界面是不现实IP地址的。另外一个vCenter Server准备就绪的表现是IP地址ping得通了。
  4. vCenter Server第一次启动后,需要打开 https://vcenter_ip:5480 继续完成vCenter Server的配置。
  5. Administrator@vsphere.local 的密码就是你在OVA导入界面里输入的密码。

2018 5月28日更新:


  1. 重启vCenter Server虚拟机。
  2. 在Photon启动界面按“e”键。
  3. 在第二行结尾加入”rw init=/bin/bash“。具体参考这里
  4. 当你看到#提示符时,运行命令”passwd“更改root密码。
  5. 运行命令”pam_tally2 –user root“检查root密码输入错了多少次。
  6. 如果输入错误次数大于1,运行命令 “pam_tally2 –user root –reset” 解锁root账号。
  7. 重启虚拟机,现在应该可以登录了。

2018 5月31日更新:

在以上步骤的第四步中,登陆后你应该会看到vCenter Server安装向导。如果你的vCenter Server只想用IP地址,请确保“System name”项填写的是IP地址。

Deploye vCenter Server Virtual Appliance on VMware Workstation

There are a lot of articles introduce how to deploy vCenter Server virtual appliance on VMware Workstation. I tried but somehow it’s failed. Following are some notes for your reference if you want to deploy vCenter Server virtual appliance on VMware Workstation real quick.

I assume you don’t have DNS or domain servers. Native DHCP services of VMware Workstation is used. You just want to use vCenter Server for some quick testings purpose, and “host-only” NIC you want to select.

  1. vCenter Server installer validates FQDN when it’s first boot up. The process fails if FQDN doesn’t work. So please make sure “Host Network Identity” is IP address of the VM when you set the OVA options.
  2. The VM is immediately booted up after importing the OVA file. But VM NIC is “disconnected” status sometimes. You have to enable the NIC in VM properties real quick.
  3. You have to wait for about 15 – 20 minutes after first boot. Console screen doesn’t show IP address before it’s fully ready. The indicator of readiness is the IP address of the VM is responding to ping.
  4. Login https://vcenter_ip:5480 to continue vCenter Server installation after the first boot is ready.
  5. The password of Administrator@vsphere.local is same as you set during importing the OVA.

Updates 28th May 2018:

Root authentication on step 4 above maybe failed. It’s caused by root account locking. Please follow the procedures below:

  1. Reboot vCenter VM.
  2. Press “e” when you see the Photon booting screen.
  3. Add “rw init=/bin/bash” to the end of the 2nd line. Refer here for detail.
  4. Run “passwd” to change root password when you see # prompt.
  5. Run “pam_tally2 –user root” to check how many failures root hits.
  6. Run “pam_tally2 –user root –reset” to unlock root if you see more than 1 in step 5.
  7. Reboot. You should be able to login root now.

Updates 31st May 2018:

You should see the installation wizard in step 4. Please make sure “System name” field is IP address if you only want to use IP for vCenter Server.

Updates 5th Sep 2018:

You may see the following error during installation.

Could not connect to VMware Directory Service via LDAP

It indicates vCenter Server FQDN doesn’t work. If you’re a home lab, you may want to add the DNS entries in the hosts file.

The older version of cis-upgrade-runner cannot be removed when upgrade vCenter Server 6.0

When you upgrade or patch vCenter Server 6.0 for Windows, you may see following symptoms:

“The older version of cis-upgrade-runner cannot be removed. Contact your technical support group.”

Or error code 1063:

“Installation of component VMware CIS upgrade runner failed with error code ‘1063’”

That means the vCenter Server installer cannot find MSI files of existing vCenter Server services. It could be following reasons:

  • You delete MSI files in “Temp” folder of the profile you used to install vCenter Server.
  • The account you used to login and install vCenter Server was roaming profile. The profile’s “Temp” folder was automatically deleted when you reboot/logoff the server.

vCenter Server 6.0 for Windows is consist of lot of standalone package. The upgrading process usually uninstall old packages, and then install newer packages. So the failure doesn’t impact to database or inventory data. You can re-initiate the upgrading again.

But you cannot manually uninstall old package since upgrading process brings down vCenter services first then uninstall old packages. If you already uninstalled old packages, the upgrading process will be stuck on bring down vCenter Services stage since some processes may already be removed. For example “vmware-python” it maps to “VMware vCenter Configuration Service”. If you manually uninstalled it before launch upgrading. It removes the service. Upgrading is not able to check status of the service.

Easiest way to get ride of this problem is

  1. Open Registry Editor (regedit) and go to the path: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products”
    You would see lot of keys are there.
  2. Search keyword “vmware-“. These keys store package info of vCenter Server.
  3. Expand one of the keys. Go to “SourceList”.
  4. The value of “LastUsedSource” is path of MSI file of old vCenter Server installer.
    For example my value is “m;1;X:\vcenter-server\packages\”.
  5. Make sure your server has the path mentioned in previous step (My case it’s X:\vcenter-server\packages\) and old MSI files are available in the path. If it’s a CD-ROM letter, you just need mount old vCenter Server image to the drive.
  6. Copy new vCenter Server image to a local folder, uncompresse and launch installer locally.
  7. Now the upgrading process can read original packages on the mentioned path in step 4. It will automatically remove old packages by the old MSI files.

There are two other workarounds. One is modify the value of “LastUsedSource” to reflect a new location of packages. But you still need the old MSI files be there. Another way is delete the key after you find it in step 2. (I never tested this way but it should work as it let vCenter Server installer thinks the server is brand new so installer can override the existing folders)

I also wrote another article for upgrading error on vCenter Server 5.5 for your reference:

CustomAction VM_InstallJRE returned actual error code 1624


Adobe Flash Player Out of Date on vSphere Web Client

You may see ‘Adobe Flash Player Out of Date’ on Chrome when you open vSphere Web Client. Click the text Chrome will update Flash Player automatically. But in some cases it doesn’t work due to maybe your Chrome is controlled by company policy or internet problem to Adobe.com. I found an article to show how to offline fix the issue. You can download Flash Player for Opera and Chromium-based browsers – PPAPI in official Adobe KB article.

You may also want to check out my other articles about Flash issue on browsers.

Flash menu appears when right click on vSphere Web Client in Chrome

Cannot open vSphere Web Client on IE11 on Windows 8.1

Flash menu appears when right click on vSphere Web Client in Chrome

There is a KB describes how to fix right click issue in IE on vSphere Web Client. But my problem was in Chrome. I searched a lot in internet but no lucky till today.

The problem was flash menu appears when I right click anything in vSphere Web Client in Chrome. I have two computers that both has Chrome installed but one has issue, other one works fine. I compared version of Chrome, noticed working one was 55.x, problematic one was 49.x. The issue gone after upgraded to 57.x.

After dig into that problem, looks like Google fixed the problem on version 54.0.2840 that there was a bug related to right click. Check out release notes here.


“No host data available” Reported in Hardware Status Tab

Just noticed a issue that nothing reported in ‘Hardware Status‘ tab of ESXi hosts in vSphere Web Client. KB 2112847 gives a solution but not works for me. The feature can be used to monitor hardware failures. I figured out a way to workaround it. You just need to login by Administrator account and click ‘Update‘ button under ‘Monitor‘ – ‘Hardware Status‘ for each ESXi host. You will get the status after few minutes.

Host Cannot Download Files From VMware vSphere Update Manager Patch Store

You may see following error when you scanning ESXi hosts by vCenter Update Manager.

Host cannot download files from VMware vSphere Update Manager patch store. Check the network connectivity and 
firewall setup, and check esxupdate logs for details.

You also see similar logs in /var/log/esxupdate.log.

[Errno -2] Name or service not known

The root cause could be following:

  1. ESXi host cannot resolve DNS name of vCenter Update Manager Server.
  2. One of the DNS servers incorrect if you set multiple DNS servers on ESXi host.

Migrate vCenter Server 5.5 Windows to 6.0 Virtual Appliance 

Virtual appliance is future of how VMware delivery their product to customers. It’s pain to migrate from vCenter Server Windows version to virtual appliance. The only way was build up new virtual appliance and move everything out of Windows vCenter Server. The challenge is you lost data if you have integrated vCenter Server with other VMware products, or using DVS.

VMware released vCenter Server Migration Tool after VMworld 2016. It gave me confidence to give it a try. I assume vCenter Server is embedded SSO. I did the migration 2 or 3 times, following is summary of my experience. The migration tool only support migrate vCenter Server 5.5 windows edition to vCenter Server 6.0 U2.


  1. vCenter Server is more like core services today since lot of 3rd party software call vCenter API to interactive with VMs. You may have some products integrated with vCenter Server already. Please upgrade to vCenter 6 compatible version before migration.
  2. I suggest create a local account on source vCenter Server if your server is domain member. You can login back source vCenter by local account in case migration failed.
  3. vCenter Server Migration Tool applies temporary IP address on destination vCenter virtual appliance during migration. It’s used to communicate with source vCenter. Please register a temporary IP address for destination vCenter Server.
  4. A helper VM is required to run migration image. Please make sure you have a free Windows VM be ready to mount migration image.
  5. SQL database is exported to source vCenter Server if you want migrate performance and event data. So you need to make sure enough space on C: drive on source vCenter. The free space should be much bigger than vCenter database size.
  6. Of course you need a vCenter Server 6 license key since old key doesn’t support the version.
  7. Some cases show migration process stopped during export SQL database. That’s because memory of source vCenter is too small. Please make sure RAM of source vCenter should be equal or greater than destination vCenter Server.
  8. The other tricky is database table. You may see migration processes is completed but destination vCenter Server doesn’t come up, and no data actually imported. That’s because ‘checksum‘ column existing in table [dbo].[VMO_ResourceElementContent] in vCenter DB. You can run following SQL query to remove it before migration.
    alter table dbo.VMO_ResourceElementContent drop column checksum;


The items above can be done anytime before the migration window. Following steps should be token during migration.

  1. You need to disable firewall and anti-virus software on old vCenter to avoid communication issue between Migration Assistant and new vCenter Server.
  2. To avoid any unstable, resource contention, or potential network connectivity lost issue, I suggest temporarily disable DRS and HA on source and destination cluster if they are virtual machine.
  3. Copy Migration Assistant from migration image to old vCenter.
  4. Take snapshot on old vCenter and backup database of old vCenter.
  5. Connect to console of source vCenter and run Migration Assistant.
  6. Mount vCenter Server 6 U2m image on helper VM. Launch vCenter migration. The migration process is straightforward. I wouldn’t introduce more here.

After Migration

Basically you need to revert all the temporary changes made before. Such as delete snapshot and DB backup, enable DRS and HA, and disable vNIC on source vCenter Server to avoid any human error.

Inventory Service无法启动

某日,vCenter Server突然无法搜索虚拟机了。在vSphere Client中搜索时会提示 Unable to connect to web services to execute query. Verify that the ‘VMware VirtualCenter Management Webservices’ service is running on https://vCenter_Server_FQDN:10443。没过几个小时用户就开始抱怨vSphere Web Client也出问题了,总是提示错误 Client is not authenticated to VMware Inventory Service – https://Inventory_Service_FQDN:10443

Continue reading

Inventory Service Cannot be Brought Up

One day, my vCenter Server suddenly lost search. It popped me “Unable to connect to web services to execute query. Verify that the ‘VMware VirtualCenter Management Webservices’ service is running on https://vCenter_Server_FQDN:10443” when I did object search on vSphere Client. Few hours later people starting complaint they got error on vSphere Web Client, it show “Client is not authenticated to VMware Inventory Service – https://Inventory_Service_FQDN:10443“.

Continue reading

Domain account locked out on vCenter Server

That’s a very small problem but it struggles you if you are enterprise datacenter administrator. As you may know the best practices to run application is by service account. But sometimes  you may testing applications by your own domain account and forget remove it.

Few days ago, my domain account locked out on domain controller. The audit report indicated it locked out by vCenter Server every 5 seconds. Then I logged in the vCenter Server, checked out Task SchedulerServicesTask Manager…etc. Nothing was running under my domain account. I stopped applications one by one on the vCenter Server and related plugin services. No help, I felt so frustrated!!!

Here is how I figured it out eventually.

  1. Download TCPView from Microsoft website.
  2. Run it on the vCenter Server.
  3. Sort by Local Address.
  4. See which foreign address is connecting the vCenter Server.

After the steps above I finally figured out that root cause was my VMware View LAB VM tried to authenticate on vCenter Server by my domain account and stored old password. I powered up the old VM few days ago.

这可能是一个很小的问题,但如果你是企业级数据中心管理员,这个问题可能会很困扰你。如你所知在日常使用中最好用Service Account来运行应用程序。但是有时候你可能和我一样需要用自己的域帐号做一些测试但之后又忘记删除了。



  1. 从微软网站下载TCPView
  2. 在vCenter服务器上运行。
  3. 选择以Local Address本地地址)排序。
  4. 查看连接到vCenter服务器的Foreign Address外部地址)。

最终原因是我几天前把一台旧的VM开机了,这台VM上是当时以我的域帐号安装的VMware View做测试用。

How to change password of vCenter Server service account

Many company use service account for vCenter Server database and services. To compliance with security policy, you may need to change password of vCenter Server at regular period. This is a way I used to change password:

1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Server/Management Webservices in Services.
3. Run vpxd.exe -p command as administrator to change database password. ( It usually located on C:\Program Files\VMware\Infrastructure\VirtualCenter Server\ )
4. RDP vCenter Server by service account.
5. Open DSN of vCenter Server and click next button to save password to DSN.
6. Reboot vCenter Server.

Notes: You maybe able to logon vCenter Server if you just restart VC services, but you will face low performance to retrieve information of host, VM…etc.

Most company may has vCenter Update Manager together with vCenter Server, the password change of vCenter Update Manager service account is similar like vCenter Server.
1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Update Manager in Services.
3. Run VMwareUpdateManagerUtility.exe as administrator. ( It’s usually on C:\Program Files (x86)\VMware\Infrastructure\Update Manager\ )
4. Input new database credential on Database Setting.
5. Re-register to vCenter Server by new credential.
6. Reboot vCenter Update Manager server.

You can also reference to http://kb.vmware.com/kb/1006482 and http://kb.vmware.com/kb/1034605.

vCenter Server Heartbeat 5.6 – Installation

I have to say you’ll not able to get what you anticipating if you follow VMware document. After referred few blogs and videos, I finally deployed the production in HA and DR mode both, it consumed a lot of time since I had to clone the VM from US to India over WAN. It’s pain, I’d like the share it to make sure you never fall in same situation.

If you don’t familiar with vCHB, please read vCenter Server Heartbeat 5.6 – Architecture.

Before install vCHB, you should know that:

  • Install vCenter Server and components on Primary Server, Secondary Server will be cloned.
  • vCenter Update Manager, vCenter Converter, ESXi Dump Collector, Syslog Collector are configured using Fully Qualified Domain Names (FQDN) rather than IP addresses.
  • Time Zone and time setting is correct.
  • Port 52267 and 57348 is enabled in firewall on both servers.
  • 2GB free memory available for vCenter Server Heartbeat.
  • Administrator right is required to install vCenter Server Heartbeat.
  • All vCenter Server components should functionally before install vCenter Server Heartbeat.
  • No * in SSO master password. ( I guess that’s a bug of 5.6U1, please refer to KB2034608 to reset master password )
  • vCenter Server FQDN is Primary Server computer name. ( It will be changed later )

Pre-configure before install vCHB:

  • Make sure Primary Server computer name is vCenter Server FQDN.
  • Change vCenter Server services to manually start up on Primary Server.
    VMware VirtualCenter Server
    VMware vSphere Profile-Drive Storage
    vCenter Inventory Service
    VMware VirtualCenter Management Webservices
  • Recovery system fingerprint encrypted file.
    Go to C:\Program Files\VMware\Infrastructure\SSOServer\utils
    Recovery footprint by following command:
    rsautil manage-secrets -a recover -m SSO Master Password
  • Power off Primary Server
  • Clone Primary Server to secondary site.
  • Disconnect vNICs on Secondary Server.
  • Power on both servers and set IP addresses.
    I use two vNICs on each server, one for Public Network, another for VMware Channel Network.
    Public Network contains two IP address, one for Management Network, another for Principle Network.
    Principle Network on both should be same if you deploy HA mode, otherwise they are different for DR mode.
  • Disable NETBIOS and DNS Register on each vNIC.
  • Leave domain and rename Secondary Server.
  • Reboot Secondary Server and connect vNICs.
  • Join Secondary Server back to domain and add proper AD groups to Administrator group.
    Note: You probably need to re-join domain twice to make sure AD synchronization correct, I got vCenter Server startup issue in initially deployment due to AD synchronization issue.
  • Create a share folder on reliable server that Primary and Secondary Server both can access.
  • Make sure configured IP addresses pingable from each server.
  • Bring up vCenter Server services on Primary Server.


  • Select Install VMware vCenter Server Heartbeat to start installation.
  • Select Primary to install vCHB on Primary Server.
  • Accept agreement.
  • Apply license key.
  • Select LAN or WAN according to your architecture.
  • Select Secondary Server is Virtual option. ( I only tested that option )
  • Confirm installation path.
  • Select vNIC for VMware Channel network.
  • Enter VMware Channel IP addresses of Primary and Secondary Server.
    For HA mode, you could use non-routable or routable IP address.
    For DR mode, you must use routable IP addresses to make sure VMware Channel network can communicate each other over WAN.
  • Select vNIC for Public Network.
  • Enter IP addresses of Principal Network for both server.
    For HA mode, IP address should be same on both server.
    For DR mode, IP addresses should be different, you have to enter manually.
    Select the options accordingly.
  • If you select Different IP addresses in step above, you will need to enter a DNS update account of Windows. ( Refer to KB1008605 if you use BIND9 DNS instead of Windows DNS service )
  • Then configure Management Network. This network is used for RDP.
  • Rename computer name of both server. It looks like only rename Primary Server, no change for Secondary Server, but you don’t have to worry about that since we already renamed Secondary Server in early step.
  • Set client port, I used default.
  • Select components you want to protect and enter vCenter Login, this Login must have Administrator right on vCenter Server.
    Also input SSO master password, please note the SSO master password may different with SSO administrator password, please make sure you enter correct password.
  • Enter the share path you created earlier, this folder will store cluster configuration information for Secondary Server installation.
  • vCHB start checking system.
  • You will lost RDP connectivity for 10 seconds during installation due to Package Filter installation.

  • Once the installation complete, you can start on Secondary Server, just make sure you select Secondary.
    All other steps is similar like Primary Server.

After Installation:

  • Startup vCHB services on Secondary Server.
  • Open vCenter Server Heartbeat Management Console.
  • Add each node by Management Network.
  • Wait a while, you will see similar screen like following screenshot.

vCenter Server Heartbeat 5.6 – Architecture

I start to use VMware workstation since 2002 or earlier, my bad memory can’t recall it. That’s 1st generation of virtualization. If you look at today’s virtual world, we are on the way to “Matrix”! J Enterprise is virtualizing more and more server lead to vCenter Server becomes to a critical role. We have to prepare for any contingency. vCenter Server Heartbeat (vCHB) is a nice candidate for protecting vCenter Server. It provides your infrastructure ability to prevent downtime/outage of vCenter Server. To gearing up for implementation in production environment, I did some testing on my LAB, the product is nice, but the document is not ideal. I’d like to share my experience, this blog also referred to my project document, please let me know if you have any idea can help me make my document ideally. Thanks in advance.

vCHB is a cluster service like Microsoft Cluster Service or any other 3rd part cluster software. The benefit of this product is you don’t have to create the cluster on RDM and your ESXi maintenance operation would become much easier. You could deploy vCHB in HA or DR mode, I’ll focus on HA mode at this moment since I haven’t tested DR mode yet.


My original LAB infrastructure contains one vCenter Server with remote SQL database server, data transmits over LAN. So my vCHB topology is one SQL database (I already have MSCS to protects SQL database server), two vCenter servers (Primary Server and Secondary Server).

vCHB uses Active-Passive for HA mode, Active Role runs protected applications, Passive Role receives changed data.

Primary Server – Original vCenter Server which I want to protect, it runs all vCenter components except outage happening.

Secondary Server – Another server of the pair, it’s Passive Role. Generally it receives change of Primary Server and takes over Active Role when outage happens.

In my LAB Active Role is Primary Server, and Passive Role is Secondary Server in most of time.


vCHB have two networks: Public Network and VMware Channel Network. You could use single NIC to run all networks or multiple NICs to separate them.

VMware Channel Network – vCHB monitors alive of each via VMware Channel Network and syncs changed data, it’s very important network.

Public Network – It contains two sub-networks: Principle Network and Management Network. Principle Network for vCHB cluster, Management Network for day-to-day operation.

Confuse? To simple it, I understand the networks like that:

VMware Channel Network – Can be private IP address or any IP address outside of the subnet of Public Network. It used for heartbeat and data transmitting.

Public Network Principle Network is IP address of Cluster DNS name, Management Network is IP address for RDP, they are in same routable subnet, but better in different prefix of IP address, please refer to KB 2004926.


No special storage requirement, but 2GB free space should be there where you want to install vCHB to. We also need a reliable share folder to store cluster data, I prefer to create share folder on a server other than vCHB servers since vCHB networks usually interrupt for few seconds during vCenter failover.

Okay, I’ll share how to install vCHB in next blog, this architecture for your reference: