Host Cannot Download Files From VMware vSphere Update Manager Patch Store

You may see following error when you scanning ESXi hosts by vCenter Update Manager.

Host cannot download files from VMware vSphere Update Manager patch store. Check the network connectivity and 
firewall setup, and check esxupdate logs for details.

You also see similar logs in /var/log/esxupdate.log.

[Errno -2] Name or service not known

The root cause could be following:

  1. ESXi host cannot resolve DNS name of vCenter Update Manager Server.
  2. One of the DNS servers incorrect if you set multiple DNS servers on ESXi host.

Migrate vCenter Server 5.5 Windows to 6.0 Virtual Appliance 

Virtual appliance is future of how VMware delivery their product to customers. It’s pain to migrate from vCenter Server Windows version to virtual appliance. The only way was build up new virtual appliance and move everything out of Windows vCenter Server. The challenge is you lost data if you have integrated vCenter Server with other VMware products, or using DVS.

VMware released vCenter Server Migration Tool after VMworld 2016. It gave me confidence to give it a try. I assume vCenter Server is embedded SSO. I did the migration 2 or 3 times, following is summary of my experience. The migration tool only support migrate vCenter Server 5.5 windows edition to vCenter Server 6.0 U2.

Prerequisites

  1. vCenter Server is more like core services today since lot of 3rd party software call vCenter API to interactive with VMs. You may have some products integrated with vCenter Server already. Please upgrade to vCenter 6 compatible version before migration.
  2. I suggest create a local account on source vCenter Server if your server is domain member. You can login back source vCenter by local account in case migration failed.
  3. vCenter Server Migration Tool applies temporary IP address on destination vCenter virtual appliance during migration. It’s used to communicate with source vCenter. Please register a temporary IP address for destination vCenter Server.
  4. A helper VM is required to run migration image. Please make sure you have a free Windows VM be ready to mount migration image.
  5. SQL database is exported to source vCenter Server if you want migrate performance and event data. So you need to make sure enough space on C: drive on source vCenter. The free space should be much bigger than vCenter database size.
  6. Of course you need a vCenter Server 6 license key since old key doesn’t support the version.
  7. Some cases show migration process stopped during export SQL database. That’s because memory of source vCenter is too small. Please make sure RAM of source vCenter should be equal or greater than destination vCenter Server.
  8. The other tricky is database table. You may see migration processes is completed but destination vCenter Server doesn’t come up, and no data actually imported. That’s because ‘checksum‘ column existing in table [dbo].[VMO_ResourceElementContent] in vCenter DB. You can run following SQL query to remove it before migration.
    alter table dbo.VMO_ResourceElementContent drop column checksum;

Procedure

The items above can be done anytime before the migration window. Following steps should be token during migration.

  1. You need to disable firewall and anti-virus software on old vCenter to avoid communication issue between Migration Assistant and new vCenter Server.
  2. To avoid any unstable, resource contention, or potential network connectivity lost issue, I suggest temporarily disable DRS and HA on source and destination cluster if they are virtual machine.
  3. Copy Migration Assistant from migration image to old vCenter.
  4. Take snapshot on old vCenter and backup database of old vCenter.
  5. Connect to console of source vCenter and run Migration Assistant.
  6. Mount vCenter Server 6 U2m image on helper VM. Launch vCenter migration. The migration process is straightforward. I wouldn’t introduce more here.

After Migration

Basically you need to revert all the temporary changes made before. Such as delete snapshot and DB backup, enable DRS and HA, and disable vNIC on source vCenter Server to avoid any human error.

Inventory Service无法启动

某日,vCenter Server突然无法搜索虚拟机了。在vSphere Client中搜索时会提示 Unable to connect to web services to execute query. Verify that the ‘VMware VirtualCenter Management Webservices’ service is running on https://vCenter_Server_FQDN:10443。没过几个小时用户就开始抱怨vSphere Web Client也出问题了,总是提示错误 Client is not authenticated to VMware Inventory Service – https://Inventory_Service_FQDN:10443

Continue reading “Inventory Service无法启动”

Inventory Service Cannot be Brought Up

One day, my vCenter Server suddenly lost search. It popped me “Unable to connect to web services to execute query. Verify that the ‘VMware VirtualCenter Management Webservices’ service is running on https://vCenter_Server_FQDN:10443” when I did object search on vSphere Client. Few hours later people starting complaint they got error on vSphere Web Client, it show “Client is not authenticated to VMware Inventory Service – https://Inventory_Service_FQDN:10443“.

Continue reading “Inventory Service Cannot be Brought Up”

Domain account locked out on vCenter Server

That’s a very small problem but it struggles you if you are enterprise datacenter administrator. As you may know the best practices to run application is by service account. But sometimes  you may testing applications by your own domain account and forget remove it.

Few days ago, my domain account locked out on domain controller. The audit report indicated it locked out by vCenter Server every 5 seconds. Then I logged in the vCenter Server, checked out Task SchedulerServicesTask Manager…etc. Nothing was running under my domain account. I stopped applications one by one on the vCenter Server and related plugin services. No help, I felt so frustrated!!!

Here is how I figured it out eventually.

  1. Download TCPView from Microsoft website.
  2. Run it on the vCenter Server.
  3. Sort by Local Address.
  4. See which foreign address is connecting the vCenter Server.

After the steps above I finally figured out that root cause was my VMware View LAB VM tried to authenticate on vCenter Server by my domain account and stored old password. I powered up the old VM few days ago.

这可能是一个很小的问题,但如果你是企业级数据中心管理员,这个问题可能会很困扰你。如你所知在日常使用中最好用Service Account来运行应用程序。但是有时候你可能和我一样需要用自己的域帐号做一些测试但之后又忘记删除了。

几天前,我的域帐号被域控制器锁定了。域报告显示我的帐号每5秒钟就会被vCenter服务器锁定一次。我在vCenter服务器上检查了任务管理器、服务、计划任务等等,并没有发现任何东西使用我的帐号。然后我将vCenter服务器上的所有服务、应用程序都停了,还是不行!

最终我找到了问题原因,以下是方法:

  1. 从微软网站下载TCPView
  2. 在vCenter服务器上运行。
  3. 选择以Local Address本地地址)排序。
  4. 查看连接到vCenter服务器的Foreign Address外部地址)。

最终原因是我几天前把一台旧的VM开机了,这台VM上是当时以我的域帐号安装的VMware View做测试用。

How to change password of vCenter Server service account

Many company use service account for vCenter Server database and services. To compliance with security policy, you may need to change password of vCenter Server at regular period. This is a way I used to change password:

1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Server/Management Webservices in Services.
3. Run vpxd.exe -p command as administrator to change database password. ( It usually located on C:Program FilesVMwareInfrastructureVirtualCenter Server )
4. RDP vCenter Server by service account.
5. Open DSN of vCenter Server and click next button to save password to DSN.
6. Reboot vCenter Server.

Notes: You maybe able to logon vCenter Server if you just restart VC services, but you will face low performance to retrieve information of host, VM…etc.

Most company may has vCenter Update Manager together with vCenter Server, the password change of vCenter Update Manager service account is similar like vCenter Server.
1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Update Manager in Services.
3. Run VMwareUpdateManagerUtility.exe as administrator. ( It’s usually on C:Program Files (x86)VMwareInfrastructureUpdate Manager )
4. Input new database credential on Database Setting.
5. Re-register to vCenter Server by new credential.
6. Reboot vCenter Update Manager server.

You can also reference to http://kb.vmware.com/kb/1006482 and http://kb.vmware.com/kb/1034605.

vCenter Server Heartbeat 5.6 – Installation

I have to say you’ll not able to get what you anticipating if you follow VMware document. After referred few blogs and videos, I finally deployed the production in HA and DR mode both, it consumed a lot of time since I had to clone the VM from US to India over WAN. It’s pain, I’d like the share it to make sure you never fall in same situation.

If you don’t familiar with vCHB, please read vCenter Server Heartbeat 5.6 – Architecture.

Before install vCHB, you should know that:

  • Install vCenter Server and components on Primary Server, Secondary Server will be cloned.
  • vCenter Update Manager, vCenter Converter, ESXi Dump Collector, Syslog Collector are configured using Fully Qualified Domain Names (FQDN) rather than IP addresses.
  • Time Zone and time setting is correct.
  • Port 52267 and 57348 is enabled in firewall on both servers.
  • 2GB free memory available for vCenter Server Heartbeat.
  • Administrator right is required to install vCenter Server Heartbeat.
  • All vCenter Server components should functionally before install vCenter Server Heartbeat.
  • No * in SSO master password. ( I guess that’s a bug of 5.6U1, please refer to KB2034608 to reset master password )
  • vCenter Server FQDN is Primary Server computer name. ( It will be changed later )

Pre-configure before install vCHB:

  • Make sure Primary Server computer name is vCenter Server FQDN.
  • Change vCenter Server services to manually start up on Primary Server.
    VMware VirtualCenter Server
    VMware vSphere Profile-Drive Storage
    vCenter Inventory Service
    VMware VirtualCenter Management Webservices
  • Recovery system fingerprint encrypted file.
    Go to C:Program FilesVMwareInfrastructureSSOServerutils
    Recovery footprint by following command:
    rsautil manage-secrets -a recover -m SSO Master Password
  • Power off Primary Server
  • Clone Primary Server to secondary site.
  • Disconnect vNICs on Secondary Server.
  • Power on both servers and set IP addresses.
    I use two vNICs on each server, one for Public Network, another for VMware Channel Network.
    Public Network contains two IP address, one for Management Network, another for Principle Network.
    Principle Network on both should be same if you deploy HA mode, otherwise they are different for DR mode.
  • Disable NETBIOS and DNS Register on each vNIC.
  • Leave domain and rename Secondary Server.
  • Reboot Secondary Server and connect vNICs.
  • Join Secondary Server back to domain and add proper AD groups to Administrator group.
    Note: You probably need to re-join domain twice to make sure AD synchronization correct, I got vCenter Server startup issue in initially deployment due to AD synchronization issue.
  • Create a share folder on reliable server that Primary and Secondary Server both can access.
  • Make sure configured IP addresses pingable from each server.
  • Bring up vCenter Server services on Primary Server.

Installation:

  • Select Install VMware vCenter Server Heartbeat to start installation.
  • Select Primary to install vCHB on Primary Server.
  • Accept agreement.
  • Apply license key.
  • Select LAN or WAN according to your architecture.
  • Select Secondary Server is Virtual option. ( I only tested that option )
  • Confirm installation path.
  • Select vNIC for VMware Channel network.
  • Enter VMware Channel IP addresses of Primary and Secondary Server.
    For HA mode, you could use non-routable or routable IP address.
    For DR mode, you must use routable IP addresses to make sure VMware Channel network can communicate each other over WAN.
  • Select vNIC for Public Network.
  • Enter IP addresses of Principal Network for both server.
    For HA mode, IP address should be same on both server.
    For DR mode, IP addresses should be different, you have to enter manually.
    Select the options accordingly.
  • If you select Different IP addresses in step above, you will need to enter a DNS update account of Windows. ( Refer to KB1008605 if you use BIND9 DNS instead of Windows DNS service )
  • Then configure Management Network. This network is used for RDP.
  • Rename computer name of both server. It looks like only rename Primary Server, no change for Secondary Server, but you don’t have to worry about that since we already renamed Secondary Server in early step.
  • Set client port, I used default.
  • Select components you want to protect and enter vCenter Login, this Login must have Administrator right on vCenter Server.
    Also input SSO master password, please note the SSO master password may different with SSO administrator password, please make sure you enter correct password.
  • Enter the share path you created earlier, this folder will store cluster configuration information for Secondary Server installation.
  • vCHB start checking system.
  • You will lost RDP connectivity for 10 seconds during installation due to Package Filter installation.

  • Once the installation complete, you can start on Secondary Server, just make sure you select Secondary.
    All other steps is similar like Primary Server.

After Installation:

  • Startup vCHB services on Secondary Server.
  • Open vCenter Server Heartbeat Management Console.
  • Add each node by Management Network.
  • Wait a while, you will see similar screen like following screenshot.