Year: 2013

vCenter Server Heartbeat 5.6 – Architecture

I start to use VMware workstation since 2002 or earlier, my bad memory can’t recall it. That’s 1st generation of virtualization. If you look at today’s virtual world, we are on the way to “Matrix”! J Enterprise is virtualizing more and more server lead to vCenter Server becomes to a critical role. We have to prepare for any contingency. vCenter Server Heartbeat (vCHB) is a nice candidate for protecting vCenter Server. It provides your infrastructure ability to prevent downtime/outage of vCenter Server. To gearing up for implementation in production environment, I did some testing on my LAB, the product is nice, but the document is not ideal. I’d like to share my experience, this blog also referred to my project document, please let me know if you have any idea can help me make my document ideally. Thanks in advance.

vCHB is a cluster service like Microsoft Cluster Service or any other 3rd part cluster software. The benefit of this product is you don’t have to create the cluster on RDM and your ESXi maintenance operation would become much easier. You could deploy vCHB in HA or DR mode, I’ll focus on HA mode at this moment since I haven’t tested DR mode yet.

Server

My original LAB infrastructure contains one vCenter Server with remote SQL database server, data transmits over LAN. So my vCHB topology is one SQL database (I already have MSCS to protects SQL database server), two vCenter servers (Primary Server and Secondary Server).

vCHB uses Active-Passive for HA mode, Active Role runs protected applications, Passive Role receives changed data.

Primary Server – Original vCenter Server which I want to protect, it runs all vCenter components except outage happening.

Secondary Server – Another server of the pair, it’s Passive Role. Generally it receives change of Primary Server and takes over Active Role when outage happens.

In my LAB Active Role is Primary Server, and Passive Role is Secondary Server in most of time.

Networking

vCHB have two networks: Public Network and VMware Channel Network. You could use single NIC to run all networks or multiple NICs to separate them.

VMware Channel Network – vCHB monitors alive of each via VMware Channel Network and syncs changed data, it’s very important network.

Public Network – It contains two sub-networks: Principle Network and Management Network. Principle Network for vCHB cluster, Management Network for day-to-day operation.

Confuse? To simple it, I understand the networks like that:

VMware Channel Network – Can be private IP address or any IP address outside of the subnet of Public Network. It used for heartbeat and data transmitting.

Public Network – Principle Network is IP address of Cluster DNS name, Management Network is IP address for RDP, they are in same routable subnet, but better in different prefix of IP address, please refer to KB 2004926.

Storage

No special storage requirement, but 2GB free space should be there where you want to install vCHB to. We also need a reliable share folder to store cluster data, I prefer to create share folder on a server other than vCHB servers since vCHB networks usually interrupt for few seconds during vCenter failover.

Okay, I’ll share how to install vCHB in next blog, this architecture for your reference:

A disk read error occurred after upgrade HW version from 3 to 9

This was a lesson and learns for me after I recovered the data back. My data was lost and no backup…

I had a virtual machine was moved from ESX 3.0 to ESXi 5.1 host long time ago. The virtual disk size show 0 and I cannot do storage migration and snapshot on the VM due to the hardware version was 3, it’s too low.

Generally I take snapshot before upgrade VM HW version, but that’s impossible on a VM of HW version 3 that running on vCenter Server 5.1. So I upgraded the VMware Tools and then VM hardware version by Update Manager. VMware Tools was successfully upgraded, but VM hardware version upgrading got error.

Then I right clicked the VM and used “Upgrade Hardware Version” option directly, it’s successfully without any prompt…finally I got “A disk read error occurred” when boot up. L

You may think it’s caused by SCSI controller since VM hardware version 3 supports IDE virtual disk and version 9 supports only SCSI virtual disk for best performance. That’s not my case. I tried several way to recover the disk, like convert the VM by convertor, mount the disk to other virtual machine, change SCSI parameter…etc.

I don’t think hardware version upgrading changes real virtual disk too much, it must be something changed on the head section of virtual disk, or description file. After consulted with Microsoft we got it fixed finally.

When I mounted the corrupted disk on other virtual machine, partition and size was recognized correctly. And disk manager also can recognize the NTFS file system. I can saw new drive appear in My Computer as well, but it show me “File or directory corrupted…” when I tried to open the drive. It more like a file system issue… it’s easy, just run following command to check any logical errors:

Chkdsk [drive letter]

Wow….a lot of error and files was listed, then I tried command:

Chkdsk /f [drive letter]

That’s real fix logical issue of disk. I could open the drive after used this command.

I mounted the drive back to the broken VM and powered on. New issue came up…Windows show me “Windows NT could not start because the below file is missing or corrupt: C:WindowsSystem32Ntoskrnl.exe”. I replaced the file but no help. The file was existed in the location, and file size was same like other VM, it’s perhaps not file issue?

Then I open VMDK file, aha….ddb.adapterType = “LegacyESX”, changed it to ddb.adapterType = “lsilogic” according to my SCSI controller set, my lovely Windows Server startup screen came back again. J

Okay, I talked too much. To summarize the fixing steps:

  • Mount the broken disk to a good virtual machine with same operating system. ( I’m not sure is it ok to mount on higher version of OS )
  • Run chkdsk [drive letter] to check if logical error existing.
  • Run chkdsk /f [drive] letter] to fix the logical error.
  • Unmounts the disk from good VM.
  • Edit the VMDK file in ESXi console.
  • Change the value of ddb.adapterType to proper SCSI controller type according to your SCSI controller setting.
  • Mount the disk back to broken VM.
  • Power on.

Here is my learning from that contingence:

  1. vCenter Server does not verify compatibility of VM hardware version during upgrading. Actually it’s not allowed to upgrade VM version from 3 to 9 directly.
  2. vCenter Server does not allowed you choose which VM hardware version you want to upgrade to, always latest.
  3. If you upgrade VM version from 3 to 9 directly, a SCSI controller will be added to the VM, value of ddb.adapterType will be changed to LegacyESX. You will not able to boot up the VM due to Windows Server 2003 does not contain proper SCSI driver.
  4. VM version upgrading looks like changes parameters of VMDK file but don’t change too much of real virtual disk, such as NTFS mapping and MBR table…etc.

Last, you may still face BSOD after use above solution since item 3 above, you have to inject the SCSI driver, please refer to KB 1005208 and KB 1006858.

Last of last…. 🙂 please take a backup of your virtual disk before you do any change!!!!!

Unable to connect to web services to execute query

It’s been a long time since last post, I was pretty busy on a storage issue, I did a lot of work with hardware vendor and VMware for this weird issue.

During our troubleshooting, I noticed a minor problem when I try search VM in vSphere Client, everytime it gave me error message “Unable to connect to web services to execute query“, it requested me “Verify that the VMware VirtualCenter Management Webservices service is running”

I tried to reboot vCenter Server, restart Management webservices and even re-installed vSphere Client, no lucky….Finally I fixed the problem by following step:

  • Stop VMware VirtualCenter Management Webservices service on vCenter Server.
  • Backup Data folder in C:Program FilesVMwareInfrastructuretomcatwebappssmsWEB-INFclassescomvmwarevimsms.
  • Remove all sms-*.db files in Data folder.
  • Restart VMware VirtualCenter Management Webservices service.

It’s simple steps to fix the problem, but this issue confused me and VMware support for a long time. This problem appeared after we upgraded vCenter Server from 5.0 to 5.1, first thing we suspected was inventory services, error message below was logged in ds.log when we searched VM.

[2013-05-25 12:04:31,995 http-nio-/0.0.0.0-10443-exec-634  INFO  com.vmware.vim.vcauthenticate.servlets.AuthenticationServlet] Sending security error because of exception : com.vmware.vim.vcauthenticate.exception.SsoUnreachableException: com.vmware.vim.dataservices.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.sso.admin.exception.InternalError: General failure.

It looks like a authentication issue, right? So we checked SSO, service account…etc. The unclearly logs lead to a wrong way. 🙂

Since nobody complained to me, I suspected that’s a client side issue, then we tried search on another purge client but same issue. We also suspected the cache of vCenter inventory, but logs didn’t evidence it is, we cannot just reset inventory cache database since that’s production environment!

Okay, I talk too much about troubleshooting process, let’s talk about the search function of vSphere, my understood is vCenter search objects by two different way: Web Client or vSphere Client. It looks like Web Client retrieve data from database or Web Client server.

vSphere Client get data from cache database. The cache database is located in vCenter Server install folder, default path is C:Program FilesVMwareInfrastructuretomcatwebappssmsWEB-INFclassescomvmwarevimsms. the cache file is actually H2 databases, it work together with Tomcat web services, sms folder contains application files of Storage Monitoring Services, it use H2 database engine v1.2.147. Please comments if you think I’m wrong.

If the H2 database incorrupt, storage monitoring services also stop working, you can find the service in Service initializing… status with warning status in vCenter Service Status node of vSphere Client.

One solution fix two issue, I like it!

 

Get specific advanced configuration of ESXi host

Storage team said the best practics of QFullSampleSize is 32, they want to check how it’s going in our environment. It’s easy to check individual host, but pretty time consuming if you want to check 300+ hosts.
Here is a one line PowerShell script to export QFullSampleSize and QFullThreshold to a csv file.

Get-VMHost | %{ $HostName=$_.Name; $HostCluster=$_.Parent; Get-VMHostAdvancedConfiguration -VMHost $_ | % { $_.getEnumerator()| ? {$_.Key -like "*QFull*"} | select Name,Value,@{N='host';E={$HostName}},@{N='Cluster';E={$HostCluster}} } } | export-csv c:qSetting.csv

 

 

 

No permission to login to vCenter Server 5.1

Today, we P2V one vCenter Server, I re-added identify source for some reason, I didn’t modified any existing domain group and ACL.
After a while I got a interesting case. User reported they got “No permission to login to vCenter Server 5.1 by vSphere Client”.
I looked into the vpxa.log of vCenter Server, it show that:

2013-05-01T11:08:01.399-05:00 [09108 error '[SSO]' opID=6e704a51] [UserDirectorySso] AcquireToken InvalidCredentialsException: Authentication failed: Authentication failed

2013-05-01T11:08:01.399-05:00 [08644 error 'authvpxdUser' opID=5469f71e] Failed to authenticate user <xxxx>

I was not 100% sure that log related to the real problem. but that’s indicated it should be something related to authentication components.
After compared working SSO with the fault SSO, I noticed Domain Alias was blank on fault SSO:

Idenfity source

Then I added a domain group on fault vCenter Server and compared the group with working vCenter Server, it’s shows format different, just like that:
Working SSO – CONTOSOTEST-GROUP
Fault SSO – CONTOSO.COMTEST-GROUP

Okay…now I know why user logging got fault. The identify source configured Domain Alias before I removed it on fault SSO, then I added identify source without Domain Alias, and thenvCenter Server used Domain name as default prefix of domain group, it lead to original domain groups format ( CONTOSOxxxx ) cannot be identified by SSO.

So I deleted the identify source and added a same source with Domain alias, problem fixed…

How to retrieve or set Path Selection Policy by vCLI

First of all, this article is nothing related to PowerCLI. 🙂

You probably know how to set Path Selection Policy (PSP) by vSphere Client, but how you can setup 100 LUNs manually? We have some script can make your life easy.

How to retrieve LUN Path Selection Policy:

esxcli storage nmp device list | egrep “Device Display Name|Path Selection Policy:”


You will get a output like that:

Device Display Name: DGC Fibre Channel Disk (naa.600601602a102e0002cdf2a2596be211)
Path Selection Policy: VMW_PSP_RR


This script help you identify which LUN is what type of policy. Here tell you what is Path Selection Policy.

Next, let’s see how to modify these LUN PSP by script:
First, you should run following script to print out command for each LUN, don’t forget change the bold text to the PSP you prefer.

esxcli storage nmp device list | awk '/^naa/{print "esxcli storage nmp device set -d "$0" -PVMW_PSP_RR" };'


Then, copy the output to notepad and remove the local disk, for example following bold NAA indicates the LUN is a local HP disk.

esxcli storage nmp device set -d naa.600601602a102e008896dda81b88e211 -P VMW_PSP_RR
esxcli storage nmp device set -d naa.600601602a102e008861b28a596be211 -P VMW_PSP_RR
esxcli storage nmp device set -d naa.600601602a102e00560d8488b456e211 -P VMW_PSP_RR
esxcli storage nmp device set -d naa.600601602a102e00c4cd2600b456e211 -P VMW_PSP_RR
esxcli storage nmp device set -d naa.600508b1001c1e987243838af4c67891 -P VMW_PSP_RR
esxcli storage nmp device set -d naa.600601602a102e008c96dda81b88e211 -P VMW_PSP_RR


Last, copy modified text back to putty session, it will run the commands one by one.

How to retrieve RDM information by PowerCLI

I worked on move RDM LUNs of Microsoft Cluster virtual machine from one iGroup to another. To make sure the moving safe, we should record RDM LUN information before migration.

We had two VMs with almost 20 RDM LUNs, it’s pretty time consume to get the information manually, I used following script to retrieve information:

$RMDinfo = Get-HardDisk -VM virtual machine name -DiskType rawPhysical

$RDMinfo | select Parent,Filename,CapacityGB,ScsiCanonicalName,Name

 

Port Groups not Work with VLAN Tag on Cisco Switch

Few weeks ago, I tried to standardize networking of a cluster, there were 4 VLANs for production virtual machines, I binded the VLANs on one virtual switch which had 4 physical vmnic.

Then I created 4 port groups with different VLAN ID, but for some reason virtual machines unreachable via some vmnics. Network team verified port channel was good.

I tried on several ESXi 5.0 hosts in the cluster, all had same problem, finally we found that’s a Cisco switch bug….you could find detail information and work around here.

HP patching error after upgrade to Update Manager 5.1

If you installed “HP ESXi 5.0 Complete Bundle Update 1.6” via Update Manager 5.0, you would be able to see storage and power sub-system shows warning on HP server, that’s because some parameters show NULL in updated HP SIM provider.

Example:

HPVC_SAController.Name="vmwControllerHPSA1",CreationClassName="HPVC_SAController"
 CreationClassName = HPVC_SAController
 Name = vmwControllerHPSA1
 PowerManagementCapabilities = (NULL)
 ResetCapability = (NULL)
 OtherDedicatedDescriptions = (NULL)
 Dedicated = (NULL)
 NameFormat = (NULL)
 TransitioningToState = 12
 AvailableRequestedStates = (NULL)
 TimeOfLastStateChange = (NULL)
 EnabledDefault = 2
 RequestedState = 12

I think HP has called back the bundle, you may see similar error message below if you already download the patch and upgrade to Update Manager 5.1 then.

VMware vSphere Update Manager had an unknown error. Check the events and log files for details.

After upgrade to Update Manager 5.1
Cannot download software packages from patch source. Check the events and the Update Manager log for download details.

After remove "data" folder in Update Manager 5.1
No way to avoid the error message except filter your baseline to exclude HP patches.

Another blogger also described same situation here.