Thoughts of VMworld 2019

This is my 7th time been VMworld. It’s a great IT conference. Meet genius people, experience latest technologies, discuss with experts, and listen to other’s ideas. Technology is changing really fast. I still remember VMworld was talking about ‘be pioneers’ few years ago. But now we have moved to cloud world.

‘Virtualization’ was the big focusing when I first time attended VMworld. Now VMware, Microsoft and Red Hat grabbed most of market shares. ‘Virtualization’ is very maturely today. I could see most of 3rd party vendors were focused on ‘performance monitor’ on VMworld 2017. I think the reason was no space on ‘virtualization’ market, but performance monitor was a big market. After two years, we could see lot of great virtualization performance products, such as vRealize Operation Managers, NetApp OCI, Uila…etc. Even open source product like Zabbix, added more support of ‘virtualization’ product. What’s the next?

When I went to Solution Exchange this year. Backup product everywhere, they have big booths and great shows out there. My view is cloud backup would be under spotlights in next few years. The reason is customers are moving from premises to cloud, or somehow leverage cloud. Data protection is a new demand. Every storage vendor plans or already published their data protection product aims to cloud backup. Such as VMware Cloud for AWS for DR. Some new innovators also provide pure cloud based backup services. Such as Clumio, it backups data from premises to AWS cloud with 0 traffic charge.

VMware released ‘Project Pacific’ on VMworld 2019. It aims to native apps and containers. It’s a evolution of vSphere. I think it will renew vSphere product lifecycle and give vSphere administrators more opportunities for next decade if it can be successfully. Think about containers can be vMotion between ESXi hosts, and HA protected…all these vSphere attributes will be part of containers. And everything is manageable under vSphere Client HTML 5 version. I believe it will be VMware’s next big thing.

I joined a session of ESXi on ARM. Looks like it’s still on very early stage. People is still discussing use case of that architecture. One thing is it can be witness node of vSAN cluster. I think the advantage of ESXi on ARM is services are running on a virtual machine on ESXi on ARM. Virtual machine is something easy to protect, recovery or program. It means services are more stable and flexible. This attribute matches mission critical product line. It maybe more usefully on manufacturing. For example a ESXi on Raspberry Pi. It’s portable, low cost, low power consumption, can be survived in hard environment. Looks like a ideal solution. Only thing is service providers may need to cover hardware, ESXi, virtual machine, guest OS and applications. The reason is ESXi on ARM need to be well tuned. I don’t think end users like to do such complex things. What they need is ‘power up, plugin and use’ when they have a small device on hands.

Few other things of vSphere. vMotion performance will be increased. The suspending time will be significantly reduced. It will big help for database virtual machine migration. VMware technical support model will be changed. Current higher level supports will be transferred to ‘Primer Support’. For me, it’s just another way to increasing support cost. 🙂

I hope VMworld will be hosted in other city next year (Looks like not possible). San Francisco downtown is not a ideal place for big conference. It’s tight and expensive. It leads to lower quality of hotel and food.

Convert Virtual Machine of VirtualBox to ESXi

My coworker want to build a virtual machine on ESXi, but vendor only support virtual appliance of Oracle VirtualBox format. VMware has a KB article to show how to “Importing Virtual Machine from Oracle VirtualBox to VMware Fusion, Workstation, or Player (2053864)“. It’s working fine. But it’s not applicable for VMware ESXi.

If you follow the guide to export .ova file and import to ESXi. It will show error below on ESXi 6.0 or later:

Issues detected with selected template. Details….No supported hardware versions among….

After couple of hours’ deep dive. I figured out a way to convert VirtualBox to ESXi. You need Oracle VirtualBox, VMware Workstation or VMware Player and VMware ESXi host.

  1. Select the virtual machine -> Go to main menu -> File -> Export appliance.
  2. Choose the virtual machine.
  3. Make sure Format is “Open virtualization format 1.0“.
  4. Export to a .ova file.
  5. Open the .ova file in VMware Workstation or VMware Player.
  6. The import of the VM maybe failed with following error. Just click Retry button it will work.
    The import failed because xxxxx did not pass OVF specification conformance or virtual hardware compliance checks.
  7. Select the virtual machine and go to main menu -> File -> Export to OVF.
  8. VMware Workstation or VMware Player generates .ovf, .mf and .vmdk files.
  9. Edit .ovf file and find the line with keyword “VirtualSystemType“.
  10. Change the value “vmx-XX” to the version lower or equal to your ESXi version.
  11. Edit .mf file and remove SHA256 value of .ovf file in first line.
    SHA256(XXXXX.ovf)= xxxxxxxxxxxxxxxxx
  12. Now it’s ready to import to VMware ESXi host.

Conclusion

This procedure is not involve any code or command. There are also couple of other ways to convert VirtualBox to ESXi by ovftool command line. I tried several ways but didn’t work. Maybe I did something wrong.

In step 10, I changed VM version in .ovf file directly. I think you can also leverage VMware Workstation or VMware Player to downgrade the virtual machine’s version in GUI. It should work as long as the version is lower than your ESXi supported VM version.

Enable Windows Update on SCCM Managed Computer

Computer cannot contact Microsoft update service to upgrade latest patches if it’s managed by SCCM policy. Sometimes we may need to keep latest patch for testing, security or other purpose, such as when you create a golden image, you always want to keep the system up to date.

I found an article “How To: Remove WSUS Settings and Restore Windows Update Defaults” talks about how to delete registry keys to workaround the restriction.

I also found actually just need to change two keys in registry if you want to enable Windows Update service temporarily.

  1. Backup registry path below.
    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate
  2. Go to the same path above.
  3. Change value of “DisableWindowsUpdateAccess” from “1” to “0”
  4. Go to following registry path.
    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU
  5. Change value of “UseWUServer” from “1” to “0”

I also wrote another post about solution of install standalone patch on Windows Server 2016 for your reference: “The update is not applicable to your computer” When Install Standalone Patch on Windows Server 2016

Cannot Open Cisco UCS KVM Console By Java

When you lunch KVM console in Cisco UCS Manager. You probably get following error message:

Unable to launch the application

Error: you can not run this program because your system deployment.config file states that an enterprise configuration file is mandatory…

This is caused by Java. There are two things you can try to fix KVM console:

  • Install Java on a directory without “space”. For example, install it on C:\java\jre7.
  • Delete Sun folder in C:\windows\. But please make a backup of the folder since it may contains some special configuration of your enterprise.

I have another blog talking about UCS KVM issue: Cisco UCS Blade Cannot Get IP Address for KVM

Validating Connection Result of Connect-VIServer

Validating connection result of Connect-VIServer is trick since VMware doesn’t use the standard way of throwing errors.

There are different error messages for different scenarios. We can leverage “Try Catch Finally” of PowerShell to testing connection result and output customized errors.

The essential format of testing vCenter connection is below. PowerShell cannot catch error message if you don’t add “-ErrorAction Stop” when connect vCenter.

try{
    connect-viserver vcenter-xxxx -ErrorAction Stop
}
catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidLogin]{
    Write-Host "Permission issue"
}
catch [VMware.VimAutomation.Sdk.Types.V1.ErrorHandling.VimException.ViServerConnectionException]{
    Write-Host "Cannot connect to vCenter Server"
}
catch
    {Write-Host "Other issue"}
finally{
    write-host "the end."
}

How to know error types of the validating connection result?

You probably have question how did I know which error types are for what? I used a very simple way that just give it a try. 🙂

# Simulating connection error
Connect-VIServer vCenter.zhengwu.org -User "xxx" -Password "yyy"

# Format output of $Error. $Error stores output of errors in PowerShell.
$error[0] | fl * -force

# You will see similar lines below
writeErrorStream      : True
PSMessageDetails      :
Exception             : VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidLogin: 7/29/2019 9:27:30 AM   Connect-VIServer                Cannot complete login due to an
                        incorrect user name or password.         ---> VMware.Vim.VimException: Cannot complete login due to an incorrect user name or password. --->
                        System.ServiceModel.FaultException`1[VimApi_67.InvalidLogin]: Cannot complete login due to an incorrect user name or password.

VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidLogin is the error type we want to find.

Conclusion

This is very simple way to validating connection result of Connect-VIServer. Looks like the result errors coming from different assemblies. For example, connection error is VMware.VimAutomation.Sdk.Types.V1.ErrorHandling.VimException.ViServerConnectionException. It’s hard to find out all possible errors. I’m looking for a official document but no lucky. Please let me know if you know where I can reference.

Bitbucket Server Integration With Visual Studio Code on Windows

There is an official Bitbucket extension for Visual Studio Code if you use Bitbucket Cloud. Bitbucket has not yet released official extension for Bitbucket Server users. Following is how to configure Visual Studio Code to use Bitbucket Server.

Before the procedure you need to collect the following information:

  • Your user name on Bitbucket Server.
  • Your email on Bitbucket Server.
  • Decide the local path to store code. (It’s c:\temp\git\ in this guide)
  • Your account has permission to modify repositories on Bitbucket Server.
  • Assume project name is “ExampleProject” on Bitbucket Server.
  • Assume repository name in the project is “ExampleRepository“.
  • Get the URL of the target repository. (It’s https://userid@bb.zhengwu.org/scm/exampleproject/examplerepository.git in this guide)
  • Make sure you have the latest Visual Studio Code installed.

Time needed: 30 minutes.

  1. Download and install Git.

    Download windows installer in Github.
    Installation is simple. Only thing is to make sure to choose “Visual Studio Code” when it asks editor integration.

  2. Configure user name and email in Git.

    Name and email should match your account information on Bitbucket server.
    Run the commands in Windows command prompt.
    $ git config --global user.name "Name"
    $ git config --global user.email "sample@zhengwu.org"

  3. Clone repository to local.

    Open a command prompt. Go to a folder you want to clone the remote “ExampleRepositiry” to. Run command:
    git clone https://userid@bb.zhengwu.org/scm/exampleproject/examplerepository.git
    You will see a sub-folder with the name “ExampleRepository” is created in local.
    A hidden folder “.git” is created in the sub-folder. It’s used to tracking changes.

  4. Open workspace in Visual Studio Code

    Bitbucket Server configuration is completed in local.
    Now open Visual Studio Code -> File -> Add folder to workspace -> Open “ExampleRepository” folder.
    Visual Studio Code is integrated with Git in-the-box. It detects the repository automatically.
    It shows “Unrack” if you create a new file in the folder.

Conclusion

This is an expressway to integrate Bitbucket Server with Visual Studio Code. I’m still new on Git. Following are some useful information.

https://www.atlassian.com/git/tutorials/install-git?section=windows

https://www.atlassian.com/git?utm_campaign=learn-git&utm_medium=in-app-help&utm_source=stash

Virtual Machine Cannot Mount ISO on System Center Virtual Machine Manager 2012 R2

A few days ago a user report to me that he cannot mounts ISO image to virtual machines on SCVMM 2012 R2. The error message is the following:

Error (2912)

An internal error has occurred trying to contact the dcahyv01.amat.com server: NO_PARAM: NO_PARAM.

 

WinRM: URL: [http://hyper01.contoso.com:5985], Verb: [INVOKE], Method: [GetError], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/microsoft/bits/BitsClientJob?JobId={89EC51A2-633C-4E06-8B09-3A84146830B5}]

The reason is the communication between SCVMM and Hyper-V servers are blocked due to certification on SCVMM application is expired. The default expiry date of SCVMM certification is 1/1/2019. The issue got fixed after renewing certificates in System Center 2012 R2 Virtual Machine Manager.

Private IP Address Routes to L3 Subnet on Dual vNIC VM

It’s not easy for me to describe the issue in one line on the title. Let me give some background here. I have 2 set of VMs. Set 1 has VM A & VM B. Set 2 has VM C & VM D. Each VM has a vNIC configured with a private IP address. VM A and VM C also have another vNIC configured with an L3 (Routable) IP address. Each set’s private IP addresses are the same. To make sure no confusion I implemented a vRouter VM for each set. The vRouter is same as VM A or VM C, it has two vNICs. One is connected to L3 network, another is connected to the private network. This way can keep the private network traffic not going outside of the set. So the both set no disturb each other when I set same private IP addresses.

Diagram

Following are IP addresses I set for each VM:

  • VM A: 192.168.0.11
  • VM B: 192.168.0.12
  • VM C: 192.168.0.11
  • VM D: 192.168.0.12

The problem is I still can get ping responding on VM A to 192.168.0.12 when I turn off VM B. I expected to see the L2 traffic goes to it own vRouter and finds VM B is offline. But tracert command shows me the traffic goes from VM A’s L3 network to vRouter of the 2nd set, and then get the answer from VM D. Looks like the L2 ping package is broadcasting on L3 network.

The issue was fixed by enabling a feature on L3 network. It called “Enforce Subnet Check for IP Learning“. Cisco changed the name to “Limit IP Learning To Subnet“. It’s a VLAN level setting. It will not allow broadcasting the private Ip traffic on an L3 network. It forces private IP traffic to go to L2 network only.

Emulex OneConnect OCe10102 on ESXi 6.0

Please refer to following post for basic troubleshooting of Emulex OneConnect.

How to Install Proper Drivers for 3rd Party Network Adapter on ESXi 5.x

I have a box uses Emulex OneConnect OCe10102 network adapters. The adapter is quite old and Emulex brand card doesn’t support ESXi 6.0. I upgraded the server to ESXi 6.0 and the Emulex adapters lost.

In the initial troubleshooting, I noticed that the adapters are still visible in BIOS. So it should be some driver level issues. I checked VMware Compatibility Guide. The model OCe10102 doesn’t support by ESXi 6.0.

If you run the following command you will still be able to see the adapters in PCI list on ESXi.

[code language="perl"]
esxcli hardware pci list
[/code]

So it indicates the adapters are not visible in ESXi since the newer Emulex driver doesn’t contain the model of the adapter in ESXi 6.0 native driver.

Then I uninstalled the native Emulex driver for ESXi 6.0 by the following command and rebooted the ESXi host.

[code language="perl"]
esxcli software vib remove -n elxnet
[/code]

The adapters still not visible after rebooting since no any drivers for Emulex adapters. Then I downloaded the Emulex drivers for ESXi 5.5 on VMware website and uploaded the “offline” package in the zip file to /tmp directory of the host. Then installed the driver by the following command:

[code language="perl"]
esxcli software vib install -d "/tmp/xxxxx.zip"
[/code]

The adapters appeared after rebooting the host.

How to Find Out Source of Domain Accounts Locking on vCenter Server

I wrote an article talk about how to find out which services lockout domain accounts on vCenter Server. It only applies to the scenarios that domain accounts very frequently lockout. Like every 1 second. If it’s minutes, it will be hard to find out as it’s manually processing.

The other way to identify source is to use vSphere Web Client. The trick was told by VMware BCS team.

Log in to vSphere Web Client. Go to the main page and the Events node. Search for “authen” you may see some error events. The real source is red text.

User CONTOSO\test-user@192.168.1.1 .....