Category Archives: English

English version of my posts.

Maximum Supported Boot Devices in Virtual Machine BIOS

Noticed a interesting limitation on VMware virtual machines. If you configure multiple SCSI controllers and distribute more than  8 virtual  disks. You may experience randomly OS boot up failure when power cycle VMs. Only last 8 disks with higher SCSI ID present in boot order settings of BIOS. You cannot choose the disks with lower SCSI ID.

You need to following up VMware KB “Changing the boot order of a virtual machine using vmx options (2011654)” to force virtual machines boot up on proper SCSI node.

Advertisements

Automatic vSphere Capacity Report in PPT

Reporting is important to management. To be a IT Pro, you may need to run regular reports for management. Some reports may be generated time consume. vRealize Operations Manager is an alternative to create customized reports. It’s a powerful product to organize data and create PDF or CSV files on scheduled intervals. I recommend have a look if you have planned to implement performance, capacity and alarm system for virtual environment.

What if budget is constrained? Is there a way to create such kind of reports? The answer is “Yes”. I worked out an automatic workflow to create the reports. I will not provide step-by-step guide in this post since it’s advanced integration of multiple products, everyone may have different way to do that. You can even create everything by script if you have strong programming skill. I’m not, I only look for the easiest way to achieve the goal.

Here is a scenario for  example: I want to run a monthly report for vSphere CPU and memory count and present to management by PowerPoint. I want to show management the historical trend of CPU and memory data. The traditional way is collect data in vCenter, organize and create charts in PowerPoint slides. So the whole workflow is: vCenter -> PowerPoint

If you want to automate the whole process you need to introduce few things more: PowerCLI, CSV and Excel. You need to develop a PowerCLI script to grab CPU and memory data on vCenter Server, then export the data to a CSV table by PowerShell command export-csv. Then import the table to an Excel file by Office feature Query Data. It loads the CSV table dynamically, you can even specific what data can be queried by filter.

Once the table is present in Excel, you need to create a chart accordingly. It’s trick when you paste the chart to PowerPoint Slide. You need to use Paste Special to paste the chart as Microsoft Excel Chart Object. The pasted chart can be updated automatically when you open the PowerPoint file.

The last step is created a scheduled task to run the PowerCLI script. Make sure you read my blog Extremely slow when run PowerShell script by scheduled tasks before create the task.

You can also configure the Excel file to automatically update table by CSV file.

Cannot Launch Patch Installer on Windows Server 2016

I was trying to update one Windows Server 2016 by standalone patch file. Somehow nothing happened after I double click the installer file. That’s because Windows Server 2016 prevent execute the  file due to it’s download from internet.

The quick fix is right click the file – Properties – Check Unblock – Click OK button.

Further more. The file has ADS (alternate data streams) attached. The ADS marked the file as download from internet.

You can run following two PowerShell commands to figure out object and value of the ADS.

PS C:\> Get-Item test file.msu -Stream *
PSPath : Microsoft.PowerShell.Core\FileSystem::C:\Users\wzheng110917a\Dow
 $DATA
PSParentPath : Microsoft.PowerShell.Core\FileSystem::C:\Users\wzheng110917a\Dow
PSChildName : 20171011_KB4038801_Updates.msu::$DATA
PSDrive : C
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : C:\Users\wzheng110917a\Downloads\20171011_KB4038801_Updates.msu
Stream : :$DATA
Length : 1241376269

PSPath : Microsoft.PowerShell.Core\FileSystem::C:\Users\wzheng110917a\Dow
 one.Identifier
PSParentPath : Microsoft.PowerShell.Core\FileSystem::C:\Users\wzheng110917a\Dow
PSChildName : 20171011_KB4038801_Updates.msu:Zone.Identifier
PSDrive : C
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : C:\Users\wzheng110917a\Downloads\20171011_KB4038801_Updates.msu
Stream : Zone.Identifier
Length : 26

 

PS C:\> Get-Content testfile.msu -Stream Zone.Identifier
[ZoneTransfer]
ZoneId=3

You can  see the ZoneId is 3. Following is a table to show which type of file it is.

0     My Computer 
1     Local Intranet Zone 
2     Trusted sites Zone 
3     Internet Zone 
4     Restricted Sites Zone

For more reference please read Microsoft blog “Alternate Data Streams in NTFS“.

You can use Unblock-File if you want to unblock multiple files.

 

Memory Errors on Modern Servers

I used to see memory degrading on  Cisco  UCS blades. But less see on HPE blades. I thought it maybe quality control problem of Cisco manufacture. Today I read two articles in Cisco website, it explains why we see memory degrading and how it works. I attached the articles below.

Managing Correctable Memory Errors on Cisco UCS Servers

UCS Enhanced Memory Error Management

The conduction in the whitepaper is not only specific for Cisco UCS, but also for any modern servers. Following is summary of why memory errors rates is going high nowadays.

  • Larger memory systems contain more bits
  • Higher capacity DRAM chips require smaller bit cells which result in fewer stored charges per bit
  • Lower operating voltages can lead to reduced noise margin
  • Higher operating speeds can lead to reduced timing margin

Oracle Utilizes 50% of Physical Processors on HPE Server

DBA team told me Oracle was running slow on a HPE server. I observed the CPU utilization was about 50% of overall capacity. Whenever Oracle database bumps up the system experienced slowness.

Further  digged into the issue, I see Oracle workload only ran on single physical processor, but the server has two processors. And the  Windows 2012 R2 resource manager show the system used Processor Group, the two physical processors were grouped out. This technology is described in Microsoft MSDN article.

To fix the issue you have to change value of “NUMA Group Size Optimization” to “Flat” in BIOS. Please refer to HPE article for detail  steps.

Detail of HPE server behavior  is documented here. Please note, the article says it impacts to ProLiant Gen9 and Intel E5-26xx v3 processors. But it actually also impacts to Intel E5-26xx v4 and Synergy blades.

Login To The Query Service Failed When Search Virtual Machine

You may see following problem if you login vCenter  Server 6.0 by vSphere Client:

Login to the query service failed.

The server could not interpret the communication from the client. (The remote server returned an error: (500) Internal Server Error.)

That’s because “Use Windows session credentials” checkbox is selected. Deselect it and give it a try.

Refer KB Searching the Inventory with the vSphere Client fails (2143566)

Virtual Machine Cannot Get DHCP IP Address

Just a quick post. When virtual machine cannot get DHCP IP address the first thing you want to check is firewall. Whatever Windows firewall or physical firewall. You should make sure UDP port 67 and 68 are not blocked. Otherwise you  will see the virtual machine gets 169.x.x.x IP address only.

The two  ports is required for DHCP client to query IP addresses. The methodology is introduced in RFC document.

DHCP uses UDP as its transport protocol. DHCP messages from a client
to a server are sent to the ‘DHCP server’ port (67), and DHCP
messages from a server to a client are sent to the ‘DHCP client’ port
(68). A server with multiple network address (e.g., a multi-homed
host) MAY use any of its network addresses in outgoing DHCP messages.

I also got some ideas in this post.

Adobe Flash Player Out of Date on vSphere Web Client

You may see ‘Adobe Flash Player Out of Date’ on Chrome when you open vSphere Web Client. Click the text Chrome will update Flash Player automatically. But in some cases it doesn’t work due to maybe your Chrome is controlled by company policy or internet problem to Adobe.com. I found an article to show how to offline fix the issue. You can download Flash Player for Opera and Chromium-based browsers – PPAPI in official Adobe KB article.

You may also want to check out my other articles about Flash issue on browsers.

Flash menu appears when right click on vSphere Web Client in Chrome

Cannot open vSphere Web Client on IE11 on Windows 8.1

Network Latency on Virtual Machine

Slight network latency may cause application problem  on sensitive virtual machines. Even the network responding time is just 3 or 7 ms. There is a way to improve the  stability of responding latency – Enable RSS on NIC.

Network traffic is handled by single CPU core when RSS is disabled. Enable it will distribute the workload to 4 cores by default. You can increase CPU for RSS by change registry.

To summarize the solution. Go to Device Manager -> NIC properties -> Advance -> Find RSS option  and enable it. You will see 2 – 3 network drops when applying it.

You can refer following articles for detail.

Poor network performance or high network latency on Windows virtual machines

Virtual Receive-side Scaling in Windows Server 2012 R2

Regarding increase CPU for RSS. Read following article to learn how to modify it.

Setting the Number of RSS Processors

vRealize Operation Manager Installation Pending on install.sh

If your company implemented firewall and blocked public NTP server, you may see installation of vRealize Operation Manager pending on ./install.sh on console. That’s because the installer tries to negotiate with NTP server http://www.iana.org. The firewall blocked the traffic.

VMware TAM Manager Shan told me  there are two options on firewall to block traffic: REJECT and DROP. REJECT means firewall responding to the request and let source device knows it’s rejected. DROP means firewall immediately ignores the request and no responding to source device. Looks like there is a bug in vROPs code that it hung if NTP request gets drop and no responding.

The workaround is create a port group without physical uplinks and install vRealize Operation  Manager. Then move it to proper network after installation  is completed. You can configure correct IP addresses when import the OVF file so later on you just need simply move the network.

Flash menu appears when right click on vSphere Web Client in Chrome

There is a KB describes how to fix right click issue in IE on vSphere Web Client. But my problem was in Chrome. I searched a lot in internet but no lucky till today.

The problem was flash menu appears when I right click anything in vSphere Web Client in Chrome. I have two computers that both has Chrome installed but one has issue, other one works fine. I compared version of Chrome, noticed working one was 55.x, problematic one was 49.x. The issue gone after upgraded to 57.x.

After dig into that problem, looks like Google fixed the problem on version 54.0.2840 that there was a bug related to right click. Check out release notes here.

 

My New Toy Raspberry Pi 3

I own Raspberry Pi 1 and 2 both. I used to do some small projects and learn Python on its. It’s great computer to learn scripting and Linux. But not something can be used in daily based.

I was excited when I hear Raspberry Pi foundation released Pi 3. Thanks foundation brought our world a powerful, cheap and flexible computer that can replace my laptop. In general, I think Pi 3 can be used for daily work in GUI interface. It comes with embedded WiFi, Bluetooth and quad-core 1.2GHz CPU! It frees up the 4 USB ports so I can use it for other purpose.

I have a laptop that motherboard was dead, screen still works. So I bought driver board for the LED screen. Connected Pi 3 to the driver board by HDMI cable. The official Pi 2 case is compatible with Pi 3 except power LED in the other side. I use the latest OS “RASPBIAN JESSIE WITH PIXEL“. I have to say the graphical interface is awesome! It’s more like a modern, real computer. I use default browser Chromium which is similar with Chrome PC version. I also installed “Google Input Tool” extension on Chromium so I can input Chinese. Since most of my work on websites so I don’t have to install input tool on OS. And I saw some interface problem when use Chinese input tool for Linux. I also tried bluetooth headset, not working well. So I turned to USB headset, it works perfectly.

I opened 10 more tabs in Chromium just like how do I use my PC. I also ran a online website to streaming music to USB headset. Wrote blog, brows photos, edit files…etc. I didn’t see any performance issue so far. I guess my high performance TF card may helped. I bought Samsung 64GB UHS-1 Class10 for my Pi 3.

 

vSphere Data Protection End of Availability

Just a quick update that VDP will discontinued in future release of vSphere. It doesn`t impact to existing customers till 2022. VMware offers free migration to Dell EMC Avamar Virtual Edition in limited time if customers want to move to other backup solution. Customers can also use other 3rd party backup solutions to integrate with vCenter Server via storage API.

Read this FAQ for detail. VMware products lifecycle matrix for your reference.

“No host data available” Reported in Hardware Status Tab

Just noticed a issue that nothing reported in ‘Hardware Status‘ tab of ESXi hosts in vSphere Web Client. KB 2112847 gives a solution but not works for me. The feature can be used to monitor hardware failures. I figured out a way to workaround it. You just need to login by Administrator account and click ‘Update‘ button under ‘Monitor‘ – ‘Hardware Status‘ for each ESXi host. You will get the status after few minutes.

Host Cannot Download Files From VMware vSphere Update Manager Patch Store

You may see following error when you scanning ESXi hosts by vCenter Update Manager.

Host cannot download files from VMware vSphere Update Manager patch store. Check the network connectivity and 
firewall setup, and check esxupdate logs for details.

You also see similar logs in /var/log/esxupdate.log.

[Errno -2] Name or service not known

The root cause could be following:

  1. ESXi host cannot resolve DNS name of vCenter Update Manager Server.
  2. One of the DNS servers incorrect if you set multiple DNS servers on ESXi host.

Migrate vCenter Server 5.5 Windows to 6.0 Virtual Appliance 

Virtual appliance is future of how VMware delivery their product to customers. It’s pain to migrate from vCenter Server Windows version to virtual appliance. The only way was build up new virtual appliance and move everything out of Windows vCenter Server. The challenge is you lost data if you have integrated vCenter Server with other VMware products, or using DVS.

VMware released vCenter Server Migration Tool after VMworld 2016. It gave me confidence to give it a try. I assume vCenter Server is embedded SSO. I did the migration 2 or 3 times, following is summary of my experience. The migration tool only support migrate vCenter Server 5.5 windows edition to vCenter Server 6.0 U2.

Prerequisites

  1. vCenter Server is more like core services today since lot of 3rd party software call vCenter API to interactive with VMs. You may have some products integrated with vCenter Server already. Please upgrade to vCenter 6 compatible version before migration.
  2. I suggest create a local account on source vCenter Server if your server is domain member. You can login back source vCenter by local account in case migration failed.
  3. vCenter Server Migration Tool applies temporary IP address on destination vCenter virtual appliance during migration. It’s used to communicate with source vCenter. Please register a temporary IP address for destination vCenter Server.
  4. A helper VM is required to run migration image. Please make sure you have a free Windows VM be ready to mount migration image.
  5. SQL database is exported to source vCenter Server if you want migrate performance and event data. So you need to make sure enough space on C: drive on source vCenter. The free space should be much bigger than vCenter database size.
  6. Of course you need a vCenter Server 6 license key since old key doesn’t support the version.
  7. Some cases show migration process stopped during export SQL database. That’s because memory of source vCenter is too small. Please make sure RAM of source vCenter should be equal or greater than destination vCenter Server.
  8. The other tricky is database table. You may see migration processes is completed but destination vCenter Server doesn’t come up, and no data actually imported. That’s because ‘checksum‘ column existing in table [dbo].[VMO_ResourceElementContent] in vCenter DB. You can run following SQL query to remove it before migration.
    alter table dbo.VMO_ResourceElementContent drop column checksum;

Procedure

The items above can be done anytime before the migration window. Following steps should be token during migration.

  1. You need to disable firewall and anti-virus software on old vCenter to avoid communication issue between Migration Assistant and new vCenter Server.
  2. To avoid any unstable, resource contention, or potential network connectivity lost issue, I suggest temporarily disable DRS and HA on source and destination cluster if they are virtual machine.
  3. Copy Migration Assistant from migration image to old vCenter.
  4. Take snapshot on old vCenter and backup database of old vCenter.
  5. Connect to console of source vCenter and run Migration Assistant.
  6. Mount vCenter Server 6 U2m image on helper VM. Launch vCenter migration. The migration process is straightforward. I wouldn’t introduce more here.

After Migration

Basically you need to revert all the temporary changes made before. Such as delete snapshot and DB backup, enable DRS and HA, and disable vNIC on source vCenter Server to avoid any human error.

How to Integrate PowerCLI 6.5 with PowerShell and PowerShell ISE

I wrote an article to introducing how to integrate PowerCLI with PowerShell and PowerShell ISE. VMware just released PowerCLI 6.5 R1, it includes lot  of new features and modules. And somehow my way doesn’t work. Following is new way to integrate PowerCLI 6.5 with PowerShell and PowerShell ISE in Windows 10.

PowerShell and PowerShell ISE both have it own $profile. So we need to do two times.

Before we start

Please make sure your PowerShell execution policy is not restricted. You can get the setting by run  following command:

Get-ExecutionPolicy

PowerShell Integration

  1. Open PowerShell window. Run following command to confirm profile file is not existing.
    Test-Path $profile

    If return is ‘False’, go to step 2.
    If return is ‘True’, Backup the file and go to step 3.

  2. Run following command if the  profile file  doesn’t existing.
    New-Item -Path $profile -type file -force | Out-Null
    
    Test-Path $profile

    The return above should be ‘True’. Profile file is created.

  3. Run following command to include VMware PowerCLI modules in  PowerShell.
    Add-Content -Path $profile -value "# Load Windows PowerShell cmdlets for managing vSphere `r`n. 'C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1'"

    The blue text above maybe different in your environment base upon where your PowerCLI is installed.

PowerShell ISE Integration

PowerShell ISE process is same to PowerShell, only different is all the operation should be completed in PowerShell ISE window.

Reboot is not required in my environment. But anyhow please reboot if you see any issue.

The processes above integrate PowerCLI 6.5 with PowerCLI and PowerCLI ISE for  current user only. If you want to integrate for all users on the machine, you need to refer to this article.

Virtual Machine Console in vCenter is Opened by VMware Workstation

If you have VMware Workstation and VMware Remote Console both installed on Windows 10 you may see vCenter virtual machine console in vSphere Web Client is opened by VMware Workstation instead of VMware Remote  Console. If you see same case your VMware Remote Console may be hijacked. I call it “hijack” since there is no easy way to revert it back.

Continue reading

Add Language Switcher in Posts Page Automatically by WordPress plugin Polylang

If you are multi-language IT Pro you may want to show articles in multiple languages just like me. I use Polylang in my blog to switch English and Chinese version. I used to add “Language Switcher” tag in my articles. It’s manual process that I had to remember the tag in somewhere, copy and past it to articles when I was writing. It’s frustration to repeat it in every article. Another problem was I have to add “Read More” tag in each article before I add “Language Switcher” tag. That’s because each post shows a switcher in home page if I don’t do that. It confuses Polylang which URL each article should goes. I want to find a perfect way to fix this problem! Here we go…

Continue reading

Report snapshots older than X days in vROps

There are several ways to reporting snapshots. You can use PowerCLI, VRTools, or even vSphere Client itself. Today I will show you how to reporting by vRealize Operations Manager (vROps) 6.x. The benefit of  vROps reports is you can schedule it by sending email with PDF & CSV reports. I found a post discuss how to reports by vROps. But it requests modify policy, it may impacts global calculation. What if your teams request different criteria for reports?

Continue reading

Power Raspberry Pi by Portable Chargers

I want to try make a remote control car by Raspberry Pi. The first question is how to power a portable Raspberry Pi? There are lot of solutions in internet. I want to find a simple, cheap and long battery life solution. Some articles introduced power bank but it overs $100 and only provides 5v output. I need some thing can provide not only 5v power for Raspberry Pi, but also more than 7v for L298N.

Continue reading

vRealize Operations Management Pack for Cisco UCS Review

Cisco UCS blade system is the best blade system I used so far. Whatever the hardware, software or support is perfect. I recommend leverage the system for primary system of virtualization. UCS blade system architecture is different with HP. I feel it more likes a network system. Fabric Interconnect (FI) modules exchange data between uplinks and internal components. IOMs on each chassis controls data routing. Architecture is complicate, but it’s powerful to manage large datacenter. Talking about large datacenter, you may have hundred chassis or blades. Data goes through FIs, IOMs and blades, you could see issues on any layer. It’s hard to find out where exactly the problem is. UCS Manager provides statistics for ports just like how Cisco does on network switches. You can show statistics of a particular port. But it doesn’t tell you when and which layer it happened. I tested Cisco UCS adapter for vRealize Operation Manager before I reviewed NetApp adapter for vRealize Operation Manager. It’s developed by same company Blue Medora. I’d like to introduce few of this product, it’s just my personal review.

Continue reading

NetApp Management Package for vRealize Operation Manager 6

vRealize Operation Manager 6 (aka vROps) is new generation of vCenter Operation Manager. I started to use vCenter Operation Manager since version 1.0. I like the idea of self-learning and dynamic threshold. But the product only monitors virtualization layer. It would be perfect if it’s able to monitor under layer storage. In large vSphere environment, virtual machines share IO capacity of datastores. If few virtual machines running high disk IO it may lead to other virtual machines get performance degrading in same storage. Think about you have 100 datastores come from a NetApp filer, and 300 virtual machines running on its. One user says their virtual machine is slow but no workload from applications end. It hard to say where the latency comes from because multiple virtual machines may share same datastore, multiple LUNs share same aggregate, and maybe same physical disks. vCenter Operation Manager provided NetApp Adapter for 5.x few years ago. But the problem was it’s too hard to associate storage objects with vSphere datastore objects.

Continue reading

Create VM on specified OU on vRA

Best practices to manage enterprise Active Directory is organizing servers by particular properties.  For example, servers maybe put into different OU by role, business group or function…etc. Following is a vRO workflow sample to automate provisioning computers in proper OUs according to user choice in vRA Service Catalog. I’ll just give brief of each step in this article, so please make sure you understand both products before read this post.

Continue reading

Convert string to OU object in vRO

When you put virtual machine to particular OU, you may refer to virtual machine properties, such as ‘server role’, ‘server group’ or ‘user group’…etc. It’s easy to set a drop-list in blueprint of vRealize Automation Center (vRA) to let users choose this kind of properties but hard to create a computer account in corresponded OU location in vRO. That’s because vRA passes most of values to vRO as strings, Active Directory workflows in vRO do not provide a way to convert string to OU.

Continue reading

Inventory Service Cannot be Brought Up

One day, my vCenter Server suddenly lost search. It popped me “Unable to connect to web services to execute query. Verify that the ‘VMware VirtualCenter Management Webservices’ service is running on https://vCenter_Server_FQDN:10443” when I did object search on vSphere Client. Few hours later people starting complaint they got error on vSphere Web Client, it show “Client is not authenticated to VMware Inventory Service – https://Inventory_Service_FQDN:10443“.

Continue reading

How to call customized vRealize Orchestrator workflows in vRealize Automation Center

You almost can do everything as long as vRealize Automation Center (aka vRA) and vRealize Orchestrator (aka vRO) are integrated. I think that’s the hard part if you are newbie like me. After reading lot of articles, I learned how it works. Following is my experience, please let me know if you see anything wrong.

把vRA和vRO结合在一起几乎可以做任何事情。如果你和我一样是新手,和uijuede 得整合这块比较难,最近阅读了一些这方面的文章,算是有所了解了。以下是我的见解,如果有问题留言给我。

Continue reading

Core mode of Windows Server 2016 TP3 – Remote Management Tip

Microsoft just released technical preview 3 of Windows Server 2016, it’s catching up VMware on SDDC. I can see a lot of enhancement in the new version. A stable hypervisor is  prerequisite of SDDC but it’s weakness of Microsoft. You have to patch and reboot frequently, some organizations even have regular reboot schedules. Microsoft introduced core mode on Windows Server 2008, it much enhanced on Windows Server 2012 R2. But Windows Server 2012 R2 aims to SBM. I didn’t think SBM organizations really need that if you compare operation complexity of core mode with GUI.

Continue reading

PCPU locked up on Cisco UCS

PCPU 20 locked up. Failed to ack TLB invalidate

Error message of the PSOD

ESXi 5.5 Update 2 is stable version, but I got PSOD on one UCS blade few days ago. It scared me since there was a big bug when I upgraded ESXi from 5.1 to 5.5 Update 1 last year(See detail ESXi 5.5 and Emulex OneConnect 10Gb NIC), it lead to dozen virtual  machines crashed over and over again.I bet I’m gonna to die if it happens again. :-)

ESXi 5.5 Update 2 算得上比较稳定的版本了,但前几天遇到一台紫屏,差点儿吓尿了。半年前从ESXi 5.1升级到ESXi5.5 Update 1时候遇到个大BUG(详情见我的文章ESXi 5.5 and Emulex OneConnect 10Gb NIC),搞得几十台几十台机器挂,这次升级再来一次估计职业生涯就此结束了。

Continue reading

How to Automate Snapshot on Virtual Machine

I always treat virtual machine snapshots like a big risk. It caused several outages in our infrastructure. Please check out Best practices for virtual machine snapshots in the VMware to understand how it impacts production.

虚拟机快照对我来说绝对是个大威胁,已经在我的生产环境里发生过好几次由此引发的故障了。如果你要了解快照对生产环境的影响可以看看:Best practices for virtual machine snapshots in the VMware

Continue reading