My IT Adventure
It’s been a while since my last post. I got an exciting issue a few days ago. I was trying to log in to a vCenter Server in Chrome. I can see the login screen and enter the credential. However, I was not able to get into the main page. vSphere Web Client was stuck on the “loading.”
The reason is the Chrome version was 50. It’s incompatible with the vSphere Web Client version.
Here are the vSphere Client requirements for your convenience.
You may see vpxd.stats.maxQueryMetrics relevant error when retrieving performance data of ESXi hosts, virtual machines, or other entities on vCenter Server by PowerCLI. The full error message:
This operation is restricted by the administrator – ‘vpxd.stats.maxQueryMetrics’. Contact your system administrator
VMware published a KB about how to fix that. Please notice that the mentioned vCenter advanced parameter config.vpxd.stats.maxQueryMetrics is case-sensitive.
The parameter is removed in vCenter Advanced Settings after upgrading to vCenter Server 7.x. You have to reconfigure it.
My home lab virtual machines are connected on private network 192.168.1.x. All internet traffic goes to a gateway server and is forwarded over the internet network adapter. The gateway server is a DHCP server also to assign IP addresses to my lab devices. The benefit to use my own DHCP server is I can control some advanced DHCP settings in the simple GUI interface. However, I didn’t find the DHCP settings on VMware Fusion 12 for the private network. And my VMs got 172.x.x.x DHCP IP addresses automatically.
The reason is I’m using macOS Big Sur. Apple requests developers to use the vmnet framework for network virtualization programming. The framework only offers limited functions for virtualized networking. This limitation is confirmed by VMware staff in the community.
Since macOS has its own DHCP server. One workaround is to disable the native DHCP server by following the command which is mentioned in the same post:
sudo /bin/launchctl unload -w /System/Library/LaunchDaemons/bootps.plist
I installed a macOS virtual machine on VMware Fusion. As you may know, the installation time is super long. It took me about 1 hour to complete the installation. To save recovery time in case my computer OS needs to be re-installed. I exported the virtual machine to an OVA file to another partition.
However, the virtual machine failed to boot up when I imported the OVA file as a new virtual machine. I got the following error message:
Mac OS X… No Media
And the virtual machine entered the Boot Manager screen after few minutes. Mac OS X was in the boot list. However, it wouldn’t allow me to boot even I chosen it manually.
I tried to reset NVRAM for the virtual machine. It’s no luck. BTW, Apple has an official article talks about how to reset NVRAM or PRAM on your Mac. I’m not sure is it suitable for macOS virtual machines. I just simply delete the *.nvram file in the virtual machine’s package.
I also tried to reset BIOS and EFI booting order…etc. No one works.
Finally, I created a new empty virtual machine with macOS 11.0 as the operating system. And compared the configuration file between the empty and imported virtual machines. And I noticed that the guestos parameter was different:
My imported virtual machine is guestos = "other".
The empty virtual machine is guestOS = "darwin20-64".
The fix is just simply to change the guest OS from other to darwin20-64.
I’m not sure why the parameter was changed after exporting and importing. It may be a VMware Fusion bug.
VMware Fusion is quite different from VMware Workstation. To open the VMX file for a virtual machine. You need to right-click the virtual machine in the virtual machine library and hold the Option (Alt) key, then select Open Config File in Editor.
You may need to access the internet with a VPN for privacy or performance. The most popular VPN solution is Shadowsocks. If you are an IT Pro, you may face a similar issue that I’m facing: How does VM share host VPN connection?
My home lab is running on top of VMware Workstation on a Windows 10 computer. The virtual machines use the NAT network to communicate outside. It goes over the internet through my ISP directly. The performance of AWS or Azure is slow.
I tried Shadowsocks on the Windows 10 computer. The performance is good. But Shadowsocks is not like other VPN solutions. Its only functions for the host computer (The Windows 10 computer). I had to run Shadowsocks client on each virtual machine to get the better speed.
I read an article that is talking about how to share host VPN with VirtualBox. It gave me some ideas. The solution is actully similar:
You don’t have to disable the firewall on the host or guest machine. This is a very straightforward configuration to allow VM share host VPN connection.
It’s useful to get email notification when there is an alarm raised in vCenter Server. To configure the function. You need a valid SMTP server. And configure the SMTP server on vCenter Server. If there is a problem that vCenter Server is not sending email alerts. We need to isolate is it a vCenter Server side issue, or an SMTP server problem. Therefore how to identify problem source is important.
VMware moved vCenter Server to a customized Linux based platform since 5.5. It’s easy for administration, but limited ability in troubleshooting. You may notice that the telnet command is not available when troubleshooting the SMTP issue.
The following is how to troubleshoot such an issue on the vCenter Server virtual appliance.
Firstly, you need to ping the IP address and FQDN of the SMTP server. Make sure the basic connectivity between the vCenter Server and SMTP server is working.
Second, try to reach out to SMTP service on the vCenter Server to make sure the SMTP protocol can be established. Since the telnet command is unavailable on the vCenter Server virtual appliance. We use curl command as a workaround:
curl -v telnet://SMTP SERVER:SMTP PORT
For example:
curl -v telnet://zhengwu.org:25
You should see the keyword “Connected to zhengwu.org….” if the connection is successful. That means the basic connectivity between the vCenter Server and the SMTP server is working.
Then you can try sending an email manually on the vCenter Server shell. The following are the commands. Please note there is a . at the end of the command. It indicates the end of the mail.
MAIL FROM:sender@zhengwu.org RCPT TO:Reception@zhengwu.org DATA Subject: This is a testing mail This is testing content .
If SMTP is working fine. You should see following message:
250 2.0.0 xxxxxx Message accepted for delivery
If all of these testing are passed. Then it maybe indicate it’s a SMTP server issue.
Following are some articles for your reference:
Emails are not sent from vCenter Server Appliance 6.5 or 6.7 after an Alert/Event is triggered.
Diagnosing why VMware vCenter Server is not sending email alerts
I just upgraded a testing machine to Windows 10 1909. And installed VMware Workstation 15 latest version on the testing machine. Somehow the mouse jumps around in VMware Workstation when the mouse pointer is grabbed by a VM. This is very annoying. I only observed this issue when run VMware Workstation in remote desktop session. I didn’t see the same issue when I run a VM on a local computer.
The reason is the testing machine has WDDM graphic driver installed. You can disable WDDM graphic driver in remote desktop session by modifying group policy.
gpedit.msc
Please refer to following VMware KB to understand more about WDDM.
WDDM and XPDM graphics driver support with ESX, ESXi, Workstation and Fusion (1016770)
It’s not clear why the WDDM driver caused the problem. I think the reason probably because WDDM gives benefit to use software and hardware acceleration both. Maybe my testing VM’s physical host doesn’t have hardware acceleration capability. But the mouse jumps around in VMware Workstation due to the driver switching between the acceleration capabilities.
I also posted few articles regarding mouse issue in virtualization environment for your reference.
I worked on a interesting case few month back. A ESXi blade was not able to bring up due to management IP address didn’t responding to ping. We tried to reconfigure IP address, re-acknowledge blade, rebuild the network, and even replaced the motherboard. It was no lucky. Eventually we figured it out that another ESXi host’s management network somehow configured same MAC address. It caused the MAC address conflict on network.
This guide will show you some tips of how to troubleshooting MAC address conflicts on ESXi and Cisco UCS level.
The first article you should read is “vmk0 management network MAC address is not updated when NIC card is replaced or vmkernel has duplicate MAC address”. It helps you understand why vmkernel MAC address is not updated. The solution in the KB is change MAC address manually on ESXi. Or re-create management network.
But the reality is we usually don’t know where the conflict comes from. We only know this Cisco UCS blade installed ESXi and it doesn’t responding to ping. So you may suspect it’s a hardware issue like me.
There are some ways to check MAC address conflicts on Cisco UCS.
# Login to UCS Manager
# Run following command to show mac address status.
show platform fwm info mac <mac address> <vlan id>
# Sample
show platform fwm info mac 0025.0050.11.11 141
Login to UCS Manager GUI to generate support log.
Admins -> AllFaults, Events and Audit -> Log -> TechSupport Files
Generate a ucsm log bundle. Download and extract it. There are two major files in the log bundle: UCSM_A_TechSupport.tar.gz and UCSM_B_TechSupport.tar.gz. The files correspond to their respective Fabric Interconnect.
MAC address conflicts usually occurred on one Fabric Interconnect. So you may need to check both of them. I use A side as sample. Go to extract folder -> UCSM_A_TechSupport -> sw_trace_logs -> fwm_trace_log.current
Search keyword “REGMAC seen on border port” in the log. You need to repeat same in the log of the other FI. If you can find the entries and time is recently. Then it indicate there is conflict on the MAC address outside the UCS domain.
There maybe other reasons can cause mac address issue. I wrote in Error: No NIC found with MAC address…
Quick post. You may see “Session data container is missing. It must’ve been destroyed, probably due to logout.” after rebooting a new joined domain vCenter PSC server. It usually occurs when you login to PSC web client.
The reason is stale cookies on client browser. Clean up cookies will fix the issue.
This is my 7th time been VMworld. It’s a great IT conference. Meet genius people, experience latest technologies, discuss with experts, and listen to other’s ideas. Technology is changing really fast. I still remember VMworld was talking about ‘be pioneers’ few years ago. But now we have moved to cloud world.
‘Virtualization’ was the big focusing when I first time attended VMworld. Now VMware, Microsoft and Red Hat grabbed most of market shares. ‘Virtualization’ is very maturely today. I could see most of 3rd party vendors were focused on ‘performance monitor’ on VMworld 2017. I think the reason was no space on ‘virtualization’ market, but performance monitor was a big market. After two years, we could see lot of great virtualization performance products, such as vRealize Operation Managers, NetApp OCI, Uila…etc. Even open source product like Zabbix, added more support of ‘virtualization’ product. What’s the next?
When I went to Solution Exchange this year. Backup product everywhere, they have big booths and great shows out there. My view is cloud backup would be under spotlights in next few years. The reason is customers are moving from premises to cloud, or somehow leverage cloud. Data protection is a new demand. Every storage vendor plans or already published their data protection product aims to cloud backup. Such as VMware Cloud for AWS for DR. Some new innovators also provide pure cloud based backup services. Such as Clumio, it backups data from premises to AWS cloud with 0 traffic charge.
VMware released ‘Project Pacific’ on VMworld 2019. It aims to native apps and containers. It’s a evolution of vSphere. I think it will renew vSphere product lifecycle and give vSphere administrators more opportunities for next decade if it can be successfully. Think about containers can be vMotion between ESXi hosts, and HA protected…all these vSphere attributes will be part of containers. And everything is manageable under vSphere Client HTML 5 version. I believe it will be VMware’s next big thing.
I joined a session of ESXi on ARM. Looks like it’s still on very early stage. People is still discussing use case of that architecture. One thing is it can be witness node of vSAN cluster. I think the advantage of ESXi on ARM is services are running on a virtual machine on ESXi on ARM. Virtual machine is something easy to protect, recovery or program. It means services are more stable and flexible. This attribute matches mission critical product line. It maybe more usefully on manufacturing. For example a ESXi on Raspberry Pi. It’s portable, low cost, low power consumption, can be survived in hard environment. Looks like a ideal solution. Only thing is service providers may need to cover hardware, ESXi, virtual machine, guest OS and applications. The reason is ESXi on ARM need to be well tuned. I don’t think end users like to do such complex things. What they need is ‘power up, plugin and use’ when they have a small device on hands.
Few other things of vSphere. vMotion performance will be increased. The suspending time will be significantly reduced. It will big help for database virtual machine migration. VMware technical support model will be changed. Current higher level supports will be transferred to ‘Primer Support’. For me, it’s just another way to increasing support cost. 🙂
I hope VMworld will be hosted in other city next year (Looks like not possible). San Francisco downtown is not a ideal place for big conference. It’s tight and expensive. It leads to lower quality of hotel and food.