Error: No NIC found with MAC address…

Your HP server may runs fine on ESXi 4.x or 5.0, but you may get error message No NIC found with MAC address xx:xx:xx:xx:xx:xx after upgrade to ESXi 5.1 or later.

That’s caused by network adapter firmware, you have to upgrade server network adapter firmware by HP SPP 2013.02 or later. I would recommend you upgrade firmware of each component to this version, it’s pretty stable to run ESXi 5.1.

IPv6 link in NetApp SMVI backup log

NetApp Virtual Storage Console is my favorite to manage and backup data on NetApp attached ESXi host, there is lot of benefits to secure VM data more efficient.

The installation is pretty simple, and very less resource it requires, you can even install it on a multi-role virtual machine. But the first headache maybe the backup log.…

The default report URL is IPv6 in NetApp Virtual Storage Console. You have to add parameter in wrapper.conf file manually. Here is detail steps:

This procedure has to be repeated after NetApp Virtual Storage Console is upgraded.

1) Shut down SMVI server (via Windows service).

2) Open the wrapper.conf in C:Program FilesNetAppVirtual Storage Consolesmviserveretc

3) Locate section

 

Java Additional Parameters
wrapper.java.additional.1=-XX:MaxPermSize=128m
wrapper.java.additional.2=-Dcom.sun.management.jmxremote
wrapper.java.additional.3=-Dcommon.dir=.
wrapper.java.additional.4=-Dorg.apache.cxf.Logger=org.apache.cxf.common.logging.Log4jLogger

4) Add following line:

wrapper.java.additional.5=-Djava.net.preferIPv4Stack=true

5) Start SMVI server (via Windows service).

I need some change in my life

大约在10年前,我参加过一个CCNA的培训,认识了当时给我们上课的老师——杨佳,那时候他已经是CCIE了。说到CCIE、CCNA,就要先说说Cisco(思科)了。网络是通过路由器、交换机连接在一起的,这个地球上60%以上路由器、交换机是思科制造,可以这么说,思科创造并提供着网络世界。CCIE是思科公司产品专家的最高级别认证,每年只有很少的人可以通过这个考试,因为考试内容几乎全部是上机实战,很少一选择填空,而且是全英文的,一旦拿到CCIE意味着你掌握了非常丰富的网络知识,可以真的称得上网络专家了。我的老师杨佳就是其中一名。

Today, I’m in company shutdown, have few spare time. I read his blog http://www.yangjia.org/, pretty interesting, look at his colorful life, come back to my life. That’s too much different! I admit single life is different with marriaged life, but that’s the life, the freedom I wanna to be! At least, I need some change in my life…

Hello world Can I start?!

It’s be a while. Provisioning, formatting, and re-provisioning. Compare each Linux distribution, finally I decide to use Debian, since my Raspberry Pi official image also base upon Debian. I also registered this brand new domain to re-start my blog life.

都不知道是第几次Hello world了,这次一定要开始了,先把之前在Wordpres的文章搬过来再说!

Unable to load status of objects in vCenter Server 5.1

On today’s troubleshooting, I faced a very weird problem. vCenter Server services were up and running fine, it’s able to connect by vSphere client, but VMs, hosts show gray, and I cannot power on VM via PowerCLI.

After went through each components of vCenter Server, I noticed the database size was 230GB, but only ~20 hosts were there. So I asked DBA team truncat event tables and shrink database. Issue gone after database optimization.

You may want to do same if you face similar issue.

How to change password of vCenter Server service account

Many company use service account for vCenter Server database and services. To compliance with security policy, you may need to change password of vCenter Server at regular period. This is a way I used to change password:

1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Server/Management Webservices in Services.
3. Run vpxd.exe -p command as administrator to change database password. ( It usually located on C:Program FilesVMwareInfrastructureVirtualCenter Server )
4. RDP vCenter Server by service account.
5. Open DSN of vCenter Server and click next button to save password to DSN.
6. Reboot vCenter Server.

Notes: You maybe able to logon vCenter Server if you just restart VC services, but you will face low performance to retrieve information of host, VM…etc.

Most company may has vCenter Update Manager together with vCenter Server, the password change of vCenter Update Manager service account is similar like vCenter Server.
1. Change the password of service account of vCenter Server and database in AD.
2. Change the password of Log-On As account of vCenter Update Manager in Services.
3. Run VMwareUpdateManagerUtility.exe as administrator. ( It’s usually on C:Program Files (x86)VMwareInfrastructureUpdate Manager )
4. Input new database credential on Database Setting.
5. Re-register to vCenter Server by new credential.
6. Reboot vCenter Update Manager server.

You can also reference to http://kb.vmware.com/kb/1006482 and http://kb.vmware.com/kb/1034605.

How to find which ESXi 5.1 host lock the VM

Sometimes VM may show unknown, invalid or orphan on vCenter Server, but it still running somewhere. Some technical support engineer may request reboot VM/ESXi host, or search on each host one by one.

Declare: This article only apply to ESXi 5.1, I haven’t tested on other version.

This is easiest way to find out which host lock the VM:

  1. SSH to any host on the cluster.
  2. Go to VM folder. ( Usually it’s under /vmfs/volumes/… )
  3. Run command:  vmkfstools -D “vmx file name” | grep owner
  4. Return line similar like this:
    gen 483, mode 1, owner 529495c4-0b6a7d90-a0f3-0025b541a0dc mtime 211436
  5. The red highlight section is MAC address of owner host.
  6. Run command: esxcfg-nics -l on each ESXi host to see which host match this MAC address.

Then you need to remove the invalid VM from inventory, and login to the owner host by vSphere Client and import the VMX file again.

This procedure can save lot of time to find the real owner host, but it still consumes time if it’s a large cluster. You want to more fast? It’s possible!

After you find the MAC address, change it to regular format, like: xx:xx:xx:xx:xx:xx.

Logon vMA console and connect to vCenter Server by command: vifptarget -s vCenter Server Name

Run command: esxcfg-nics -h ESXi host name -l | grep xx:xx:xx:xx:xx:xx

More fast?

Try use Excel to list commands with all ESXi host name then past on console….

ESXi 5.1 shows 0 value for CPU/memory in vcenter

It’s been a month, i was busy to make our environment more stable, a lot of troubleshooting, webex session and discussing. Few days ago I noticed random VMs kept vMotion constantly. Some VMs got strange situation, show orphan, invalid or unknown status, but still online.

I couldn’t find any evidence why the VMs went to these status. One more thing I noticed was CPU and memory utilization of ESXi 5.1 shows 0 on vCenter server 5.1.

Following statement is not mature conclusion, it’s my inference according to DRS, HA and that particular 0 value CPU/memory. I also discussed that with VMware BCS support.

VM changed to abnormal status due to vMotion interrupted by something, more like HA kicked off due to network/storage intermittent failed. That become high chance since  DRS kept try move heavy workload VM to 0 CPU/memory host.

You have to upgrade to ESXi 5.1 latest version or vCenter Server 5.1 update 1c to permanent fix this problem.

Workaround:

Choose one option from following options, that’s temporary solution, issue will present again.

1. Restart ESXi management agent.

2. Disconnect/reconnect ESXi on vSphere client.

Update: you have to upgrade ESXi host and vcenter server both to permanent fix the problem.

HA for DMZ ESXi 5.1 cluster

Virtualization becomes popular than never this year, I see many company is transforming internal infrastructure into virtual platform.

HA is key feature of vSphere ESXi 5.1, you have to consider this part on every design, especially DMZ virtual machine.

Most DMZ ESXi cluster has restricted networking policy, even ICMP maybe not allowed. As you may know, HA detects ESXi host alive by two parts: Storage and Network.

If host can see shared storage, it means host alive.

If host can ping default gateway, it means host alive.

What if ping is disabled on default gateway? You’ll get “vSphere HA agent on this host could not reach isolation address: xxx.xxx.xxx.xxx” on each host.

It can lead to VM lost HA protection sometimes, you could use following way to fix this problem.

  1. Login to each host by SSH.
  2. Run command “vmkping xxx.xxx.xxx.xxx” to ping any ICMP enabled IP address from vmkernal ports.
  3. Record ping worked IP addresses.
  4. Right click ESXi 5.1 cluster.
  5. Edit SettingvSphere HAAdvanced Options
  6. Add das.isolationAddressX, value is the IP address of step 3, X start from 0 to 9.
  7. Repeat step 6 to add all favored IP addresses.
  8. Add das.useDefaultIsolationAddress, value is false.
  9. Right click each host and select Reconfigure for vSphere HA.