vRealize Operation Manager Installation Pending on install.sh

If your company implemented firewall and blocked public NTP server, you may see installation of vRealize Operation Manager pending on ./install.sh on console. That’s because the installer tries to negotiate with NTP server www.iana.org. The firewall blocked the traffic.

VMware TAM Manager Shan told me  there are two options on firewall to block traffic: REJECT and DROP. REJECT means firewall responding to the request and let source device knows it’s rejected. DROP means firewall immediately ignores the request and no responding to source device. Looks like there is a bug in vROPs code that it hung if NTP request gets drop and no responding.

The workaround is create a port group without physical uplinks and install vRealize Operation  Manager. Then move it to proper network after installation  is completed. You can configure correct IP addresses when import the OVF file so later on you just need simply move the network.

Blue Screen with Bug Check 50 on ESXi 5.x

Some critical VMs got blue screen in last few weeks. After working with OS and hardware vendor, we figured out the root cause eventually. It’s a CPU problem related to Intel v2 CPU of E3, E5 and E7 families. The detail information is documented in VMware KB Windows 2008 R2 and Solaris 10 64-bit virtual machines blue screen or kernel panic when running on ESXi 5.x with an Intel E5 v2 series processor.

Continue reading “Blue Screen with Bug Check 50 on ESXi 5.x”

Receive Side Scaling on UCS Blades

To implement enterprise application like SAP, Oracle or SQL on UCS virtualization environment. Default setting of UCS blades may not suitable for the application. We always expect highest performance by optimize hardware and ESXi. In my UCS training session, I noticed one “hidden” parameter may helpful for performance.

Receive Side Scaling – So called RSS, it’s a feature that allows you to utilize multiple CPUs and multiple cores per CPU to process the receiving network load. Without RSS, all of the receive network traffic is processed by one CPU and by only one core of the CPU. Essentially, RSS distributes receiving network load to all of the CPUs and their cores.

The parameter is an option in BIOS, but it’s not under BIOS policy in UCS Manager. You should go to Servers tab, extend Policies node, and check an Eth Adapter Policy under Adapter Policy node, Receive Side Scaling (RSS) is available in Options section of right frame. Blade should be rebooted to leverage the option.

Please keep in mind that do not enable RSS if your adapters more than your CPUs, it will cause unexpected network transmit failed. RSS option must be enabled on UCS policy before enable it on OS layer (I confirmed with Cisco TAC, is that true?). Regarding OS layer, please refer to those articles.

Receive side scaling on Intel® Network Adapters

How to enable Receive Side Scaling on Microsoft Windows Server 2008 R2

You don’t have to enable the option if network traffic is not a concern.

Error: No NIC found with MAC address…

Your HP server may runs fine on ESXi 4.x or 5.0, but you may get error message No NIC found with MAC address xx:xx:xx:xx:xx:xx after upgrade to ESXi 5.1 or later.

That’s caused by network adapter firmware, you have to upgrade server network adapter firmware by HP SPP 2013.02 or later. I would recommend you upgrade firmware of each component to this version, it’s pretty stable to run ESXi 5.1.

Port Groups not Work with VLAN Tag on Cisco Switch

Few weeks ago, I tried to standardize networking of a cluster, there were 4 VLANs for production virtual machines, I binded the VLANs on one virtual switch which had 4 physical vmnic.

Then I created 4 port groups with different VLAN ID, but for some reason virtual machines unreachable via some vmnics. Network team verified port channel was good.

I tried on several ESXi 5.0 hosts in the cluster, all had same problem, finally we found that’s a Cisco switch bug….you could find detail information and work around here.