Convert Virtual Machine of VirtualBox to ESXi

My coworker want to build a virtual machine on ESXi, but vendor only support virtual appliance of Oracle VirtualBox format. VMware has a KB article to show how to “Importing Virtual Machine from Oracle VirtualBox to VMware Fusion, Workstation, or Player (2053864)“. It’s working fine. But it’s not applicable for VMware ESXi.

If you follow the guide to export .ova file and import to ESXi. It will show error below on ESXi 6.0 or later:

Issues detected with selected template. Details….No supported hardware versions among….

After couple of hours’ deep dive. I figured out a way to convert VirtualBox to ESXi. You need Oracle VirtualBox, VMware Workstation or VMware Player and VMware ESXi host.

  1. Select the virtual machine -> Go to main menu -> File -> Export appliance.
  2. Choose the virtual machine.
  3. Make sure Format is “Open virtualization format 1.0“.
  4. Export to a .ova file.
  5. Open the .ova file in VMware Workstation or VMware Player.
  6. The import of the VM maybe failed with following error. Just click Retry button it will work.
    The import failed because xxxxx did not pass OVF specification conformance or virtual hardware compliance checks.
  7. Select the virtual machine and go to main menu -> File -> Export to OVF.
  8. VMware Workstation or VMware Player generates .ovf, .mf and .vmdk files.
  9. Edit .ovf file and find the line with keyword “VirtualSystemType“.
  10. Change the value “vmx-XX” to the version lower or equal to your ESXi version.
  11. Edit .mf file and remove SHA256 value of .ovf file in first line.
    SHA256(XXXXX.ovf)= xxxxxxxxxxxxxxxxx
  12. Now it’s ready to import to VMware ESXi host.

Conclusion

This procedure is not involve any code or command. There are also couple of other ways to convert VirtualBox to ESXi by ovftool command line. I tried several ways but didn’t work. Maybe I did something wrong.

In step 10, I changed VM version in .ovf file directly. I think you can also leverage VMware Workstation or VMware Player to downgrade the virtual machine’s version in GUI. It should work as long as the version is lower than your ESXi supported VM version.

Private IP Address Routes to L3 Subnet on Dual vNIC VM

It’s not easy for me to describe the issue in one line on the title. Let me give some background here. I have 2 set of VMs. Set 1 has VM A & VM B. Set 2 has VM C & VM D. Each VM has a vNIC configured with a private IP address. VM A and VM C also have another vNIC configured with an L3 (Routable) IP address. Each set’s private IP addresses are the same. To make sure no confusion I implemented a vRouter VM for each set. The vRouter is same as VM A or VM C, it has two vNICs. One is connected to L3 network, another is connected to the private network. This way can keep the private network traffic not going outside of the set. So the both set no disturb each other when I set same private IP addresses.

Diagram

Following are IP addresses I set for each VM:

  • VM A: 192.168.0.11
  • VM B: 192.168.0.12
  • VM C: 192.168.0.11
  • VM D: 192.168.0.12

The problem is I still can get ping responding on VM A to 192.168.0.12 when I turn off VM B. I expected to see the L2 traffic goes to it own vRouter and finds VM B is offline. But tracert command shows me the traffic goes from VM A’s L3 network to vRouter of the 2nd set, and then get the answer from VM D. Looks like the L2 ping package is broadcasting on L3 network.

The issue was fixed by enabling a feature on L3 network. It called “Enforce Subnet Check for IP Learning“. Cisco changed the name to “Limit IP Learning To Subnet“. It’s a VLAN level setting. It will not allow broadcasting the private Ip traffic on an L3 network. It forces private IP traffic to go to L2 network only.

Emulex OneConnect OCe10102 on ESXi 6.0

Please refer to following post for basic troubleshooting of Emulex OneConnect.

How to Install Proper Drivers for 3rd Party Network Adapter on ESXi 5.x

I have a box uses Emulex OneConnect OCe10102 network adapters. The adapter is quite old and Emulex brand card doesn’t support ESXi 6.0. I upgraded the server to ESXi 6.0 and the Emulex adapters lost.

In the initial troubleshooting, I noticed that the adapters are still visible in BIOS. So it should be some driver level issues. I checked VMware Compatibility Guide. The model OCe10102 doesn’t support by ESXi 6.0.

If you run the following command you will still be able to see the adapters in PCI list on ESXi.

[code language="perl"]
esxcli hardware pci list
[/code]

So it indicates the adapters are not visible in ESXi since the newer Emulex driver doesn’t contain the model of the adapter in ESXi 6.0 native driver.

Then I uninstalled the native Emulex driver for ESXi 6.0 by the following command and rebooted the ESXi host.

[code language="perl"]
esxcli software vib remove -n elxnet
[/code]

The adapters still not visible after rebooting since no any drivers for Emulex adapters. Then I downloaded the Emulex drivers for ESXi 5.5 on VMware website and uploaded the “offline” package in the zip file to /tmp directory of the host. Then installed the driver by the following command:

[code language="perl"]
esxcli software vib install -d "/tmp/xxxxx.zip"
[/code]

The adapters appeared after rebooting the host.

Troubleshooting Network Performance of Virtual Machine

There are several layers of networking on the virtualization infrastructure. Guest operating system, Virtual Machine, ESXi driver, physical network adapters, RJ45/SFP and network switches…etc. Sometimes it’s hard to say where exactly caused a problem. Especially hardware layer problems. Today I worked on a very interesting case, it may give some ideas to troubleshooting network performance issue which is caused by hardware layers.

A user told me he was bothered by network performance of a virtual machine. It’s slow to copy data to NFS share. But responding to “ping” command looked good. I didn’t see any issue on virtual machine layer. VMware Tools was up to date, Windows OS was patched, virtual network adapter type was VMXNET3 and VM version was also up to date.

When I tried to copy an image file to share folder of the virtual machine, I did see sometimes speed was fast, but sometimes not. Since I have two physical uplinks, it led me to guess it could be one of the uplinks.

After a lot of swapping and cable changing, we eventually figured out there was a bad SFP on network switch end. I was able to observe the issue by using “psping.exe” of Microsoft Sysinternals. I used the following command to send the different size of ping package to the virtual machine. Network drops were increasing when I increased package size.

psping.exe -l <size of package> <Destination>
Example: psping.exe -l 4k xxxx.contoso.com

The size could be 1k, 2m or even larger. I think this is a good way to identify problem outside of ESXi. Especially SFP problem as such kind of problem didn’t give any CRC or error count on network switch level.

You can also use Windows native command “ping.exe” as following. The size unit is “bytes”. For example, you need to input 4096 if you want to send 4kb.

ping.exe -l <size> <Destination>
Example: ping.exe -l 4096 xxx.contoso.com

 

 

Cannot Complete File Creation Operation When Storage vMotion

Just quick notes. I saw following error  when do storage vMotion.

Cannot Complete File Creation Operation.

When check /var/log/hostd.log. I saw following errors:

2017-11-28T02:51:04.476Z info hostd[76A80B70] [Originator@6876 sub=Vimsvc.TaskManager opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Task Created : haTask--vim.host.OperationCleanup
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] CopyFromEntry: Hostlog_Dump: Hostlog /vmfs/volumes/598700ee-ec
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] UUID: 28dbb1b5-a9d8-e311-1061-03300000002d
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] MigID: 1511837464286041
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] HLState: none
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] ToFrom: none
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] MigType: invalid
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] OpType: nfc
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] WorldID: 0
2017-11-28T02:51:04.478Z warning hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Hostlog_Flush: Failed to open hostlog /vmfs/volumes/598700e
2017-11-28T02:51:04.478Z warning hostd[772C2B70] [Originator@6876 sub=Vcsvc.OCM opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] PersistToDisk: failed to persist entry /vmfs/volumes/5
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Default opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] AdapterServer caught exception: vim.fault.CannotCreateFile
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Vimsvc.TaskManager opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Task Completed : haTask--vim.host.OperationClean
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Activation [N5Vmomi10ActivationE:0x75395c80] : Invoke do
2017-11-28T02:51:04.478Z verbose hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Arg entry:
--> (vim.host.OperationCleanupManager.OperationEntry) {
--> hlogFile = "/vmfs/volumes/598700ee-ec0f9918-5b56-000000000000/XXX-VM-01/XXX-VM-01-375f29ae.hlog",
--> opId = 1511837464286041,
--> opState = "running",
--> opActivity = "nfc",
--> curHostUuid = "28dbb1b5-a9d8-e311-1061-03300000002d",
--> }
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Throw vim.fault.CannotCreateFile
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contoso\testuser] Result:
--> (vim.fault.CannotCreateFile) {
--> faultCause = (vmodl.MethodFault) null,
--> file = "/vmfs/volumes/598700ee-ec0f9918-5b56-000000000000/XXX-VM-01/XXX-VM-01-375f29ae.hlog",
--> msg = ""
--> }

It indicates there is a file cannot be created during migration. Further check on VM configuration file (.vmx) I noticed following parameter existing but the file doesn’t existing.

migrate.hostlog = "XXX-VM-01-375f29ae.hlog"

You cannot create the file directly. Workaround is create a .hlog file with other name then rename it to the same name.

BTW, there is a bug on ESXi 6.0 U1 for similar issue, but I saw this problem  on  U2. Just for your reference below.

Storage migration of a virtual machine with a name beginning with core fails with the error: Relocate virtual machine coreXX Cannot complete the operation because the file or folder coreXX-XXXXX.hlog already exists

Network Problems of Auto Deployed ESXi Host in LAB

I built a simple Auto Deploy environment by vSphere 6.5 on nested environment. I created virtual ESXi hosts on a physical ESXi host to do the testing. The whole configuration was smoothly, I’m impressed Auto Deploy can be implemented in few hours. One thing bothered me was networking.

New ESXi hosts cannot get IP addresses properly somehow. It’s not a single problem. The symptoms are ESXi hosts cannot get IP address, or the Configure Management Network was grayed out on console, or ESXi hosts can get IP address but no responding to ping. Just quick post my solutions here.

To fix all these problems you need to do following:

  1. Enable Promiscuous Mode on the vSwtich which is attached to nested ESXi hosts on physical ESXi hosts.
  2. (I did that on Web Client of vCenter 6.5 U1. You may see different procedure on earlier versions.) Edit the host profile of Auto DeployNetworking configurationHost port group — Highlight Management Network — The option Determine how MAC address for vmknic should be decided — Choose Use the MAC Address from which the system was PXE booted.

If you don’t do step 1, your nested ESXi hosts may not able to get DHCP IP addresses properly, or it can get IP addresses but maps to a new MAC address lead to network packages cannot be transmitted.

Nested ESXi hosts get a DHCP IP addresses when do PXE booting. The hosts get another new IP addresses when apply host profile as soon as management network is created. It could be two different IP addresses and the MAC address of management network could be a new one that not same to any of vmnics. It will be hard to trace back on network switch in real environment, so I think it’s better also to do step 2.

Update 10/25/2017 — You should choose “User must explicitly choose the policy option” in step 2 above if you have multiple NICs. The reason is DHCP IP address during PXE may be captured by random NICs. If you choose what I mentioned in step 2, you will see DHCP server may learns MAC address of a none management network NICs associated with management IP address. Please refer this article for more detail.

“No host data available” Reported in Hardware Status Tab

Just noticed a issue that nothing reported in ‘Hardware Status‘ tab of ESXi hosts in vSphere Web Client. KB 2112847 gives a solution but not works for me. The feature can be used to monitor hardware failures. I figured out a way to workaround it. You just need to login by Administrator account and click ‘Update‘ button under ‘Monitor‘ – ‘Hardware Status‘ for each ESXi host. You will get the status after few minutes.

Host Cannot Download Files From VMware vSphere Update Manager Patch Store

You may see following error when you scanning ESXi hosts by vCenter Update Manager.

Host cannot download files from VMware vSphere Update Manager patch store. Check the network connectivity and 
firewall setup, and check esxupdate logs for details.

You also see similar logs in /var/log/esxupdate.log.

[Errno -2] Name or service not known

The root cause could be following:

  1. ESXi host cannot resolve DNS name of vCenter Update Manager Server.
  2. One of the DNS servers incorrect if you set multiple DNS servers on ESXi host.