Wu's Blog

My IT Adventure

Category: English

English version of my posts.

  • ESXi Disconnects From vCenter

    If you are still using Windows 2008 for vCenter Server. You may see ESXi hosts lost connection back and forth on vCenter Server after recent Windows pathing. It’s not something like a heartbeat lost for a few seconds. ESXi can takes minutes to back online.

    You can see similar logs in vpxd.log:

    2018-08-03T09:24:23.337-04:00 error vpxd[20160] [Originator@6876 sub=HttpConnectionPool-000000] [ConnectComplete] Connect failed to <cs p:00000000200ed300, TCP:XXXXXXXXXXXXXXXX:443>; cnx: (null), error: class Vmacore::SystemException(Only one usage of each socket address (protocol/network address/port) is normally permitted)

    2018-08-03T09:24:23.337-04:00 error vpxd[06332] [Originator@6876 sub=Vmomi opID=HB-host-28@307067-1d257f9c] [VpxdClientAdapter] Got vmacore exception: Only one usage of each socket address (protocol/network address/port) is normally permitted

     

    2018-08-03T09:24:23.338-04:00 error vpxd[06332] [Originator@6876 sub=Vmomi opID=HB-host-28@307067-1d257f9c] [VpxdClientAdapter] Backtrace:

    –>

    –> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-3634793, tag: vpxd

    –> backtrace[00] vmacore.dll[0x001C599A]

    –> backtrace[01] vmacore.dll[0x0005C8BF]

    –> backtrace[02] vmacore.dll[0x0005DA0E]

    That’s because your Windows server installed one of following patches.

    July 10, 2018—KB4338818 (Monthly Rollup)

    July 10, 2018—KB4338823 (Security-only update)

    The fixes are the following.

    If you installed KB4338818, please install July 18, 2018—KB4338821 (Preview of Monthly Rollup)

    If you installed KB4338823, please install Improvements and fixes – Windows 7 Service Pack 1 and Windows Server 2008 R2 Service Pack 1 (KB4345459)

  • Basic Concepts: Linux Disks

    Disk Interface

    • IDE (ATA): Bandwidth is 133 Mbps. IOPS is ~100. The interface can connect maximum 2 disks.
    • SCSI: IOPS is ~ 150. It can connects 8 or 16 disks
      • Ultrascsi320 – 320 MB/s
      • Ultrascsi640 – 640 MB/s
    • SATA: Bandwidth is 6 Gbps. IOPS is ~150. It can connects to 8 or 16 disks.
    • SAS: Bandwidth is 6 Gbps. IOPS is ~200. It can connects to 8 or 16 disks.
    • USB: Bandwidth is 480 MB/s. IOPS is vary.

    Linux Disks

    Disk (Device) Types

    Block: Can be accessed randomly. Unit is “block”.

    Character: Can be accessed sequentially. Unit is “character”.

    Disk Files (FHS)

    Files are under ‘/dev/’. Every disk (device) is file on Linux environment.

    Device ID:

    Major: Primary device ID. To identify device type for proper drivers.

    Minor: Secondary device ID. It’s the entree of specific device of same type of device.

    Create new device:

    # mknode

    [root@centos] mknode /dev/usbtest b 100 231 
[root@centos] ll | grep test 
brw-r--r--. 1 root root 100, 231 Jul 1 11:02 usbtest

    Disk Name

    It’s assigned by ICANN.

    IDE: /dev/hd[a-z]

    SCSI/SATA/USB/SAS: /dev/sa[a-z][0-9]

    There are 3 identifications below. You also can identify disks by go to /dev/disk/by-*.

    • Device file name
    • Volume labels
    • UUIDs

    Disk Partition

    MBR: Master Boot Record. It stars on sector 0. Size is 512 bytes. It consist of 3 parts:

    • Bootloader and boot applications. Size is 446 bytes.
    • Partition tables. Maximum 4 tables. 16 bytes each. Size is 64 bytes.
    • MBR availability mark. ’55AA’ means “active”. Size is 2 bytes.

    The maximum ‘Primary‘ partition tables are 4. If you need more than 4 partitions, you need to create ‘Extend‘ partition as the last partition of the 4, then create ‘Logical partitions‘ on top of ‘Extend partition‘. Labels of the 4 are start from 1 to 4. ‘Logical partitions’ always starts from 5.

  • UI Hang on “Loading” on vRealize Operations Manager (vROPs)

    Sometimes you may see vROPs web UI hang, keep on “Loading”, or no responding to any click. It because you are using vRealize Operations Manager 6.6 or earlier version, and your computer is touch screen.

    There are two ways to fix that:

    1. Disable touch screen hardware in Device Manager. The device name usually is “HID-compliant touch screen” under “Human Interface Device“
    2. If you are using Chrome, follow the steps below:
      1. Open Chrome and browse chrome://flags.
      2. Find “Touch Events API” and disable it.

  • Regular Expression

    Regular Expression also calls Regexp. There are two classes regular expressions: Basic Regexp (BRE) and Extended Regexp (ERE). Regexp should be put in double quotation marks.

    It can be categorized to:

    Strings

    . – Can be any character.

    Sample: #cat /etc/fstab | grep "U..D"

It returns the lines contains U, D and whatever 2 characters between its.

    [] – Any specified single character.

    Sample: #cat /etc/fstab | grep "U[UAB]ID"

It returnes the lines contains U, I, D and U or A or B.

    [^] – Any character except specified single character.

    Sample: #cat /etc/fstab | grep "[^AB]ID"

It returnes the lines contains I and D but no A or B in left.

    [[:digit:]] – All digital characters.

    [[:lower:]] – All lower case characters.

    [[:upper:]] – All upper case characters.

    [[:alpha:]] – All alphabet characters.

    [[:alnum:]] – All alphnumeric characters.

    [[:space:]] – All space.

    [[:print:]] – All visible characters and space.

    [[:blank:]] – Space and tab.

    [[:punct:]] – Punctuation and symbols.

    Counting

     

    Location

     

    Grouping

     

     

  • Error 0x800f081f When Enable .Net 3.5 on Windows 10

    When you install vSphere Client 5.x on Windows 10 computer. You may see “Enable .net 3.5 failed” message. And when you try to enable .Net 3.5 on Windows 10 manually, it shows error code 0x800f081f.

    This issue occures on internet blocked or policy restricted computer. The only way to avoid that is use command line to specific .Net local path and force install it.

    1. Mount the Windows 10 ISO to your computer as a new drive.
    2. Copy the path of “xxx:sourcessxs“.
    3. Run the following command. 
      dism /online /enable-feature /featurename:netfx3 /all /limitaccess /source:xxx:sourcessxs

  • UCS Manager UI Fonts Size on 4K Screen

    Older UCS Manager uses Java application. The UI fonts could be extremely small on high DPI screen. The fix is:

    1. Go to “C:Program Files (x86)Javajre1.8.0_171bin“.
    2. Go to “Properties” of “jp2launcher.exe“.
    3. “Compatibility” tab -> “Change high DPI settings“.
    4. Check “Override high DPI scaling behavior….“.
    5. Select “System (Enhanced)” or “System“.

     

  • Deploye vCenter Server Virtual Appliance on VMware Workstation

    There are a lot of articles introduce how to deploy vCenter Server virtual appliance on VMware Workstation. I tried but somehow it’s failed. Following are some notes for your reference if you want to deploy vCenter Server virtual appliance on VMware Workstation real quick.

    I assume you don’t have DNS or domain servers. Native DHCP services of VMware Workstation is used. You just want to use vCenter Server for some quick testings purpose, and “host-only” NIC you want to select.

    1. vCenter Server installer validates FQDN when it’s first boot up. The process fails if FQDN doesn’t work. So please make sure “Host Network Identity” is IP address of the VM when you set the OVA options.
    2. The VM is immediately booted up after importing the OVA file. But VM NIC is “disconnected” status sometimes. You have to enable the NIC in VM properties real quick.
    3. You have to wait for about 15 – 20 minutes after first boot. Console screen doesn’t show IP address before it’s fully ready. The indicator of readiness is the IP address of the VM is responding to ping.
    4. Login https://vcenter_ip:5480 to continue vCenter Server installation after the first boot is ready.
    5. The password of [email protected] is same as you set during importing the OVA.

    Updates 28th May 2018:

    Root authentication on step 4 above maybe failed. It’s caused by root account locking. Please follow the procedures below:

    1. Reboot vCenter VM.
    2. Press “e” when you see the Photon booting screen.
    3. Add “rw init=/bin/bash” to the end of the 2nd line. Refer here for detail.
    4. Run “passwd” to change root password when you see # prompt.
    5. Run “pam_tally2 –user root” to check how many failures root hits.
    6. Run “pam_tally2 –user root –reset” to unlock root if you see more than 1 in step 5.
    7. Reboot. You should be able to login root now.

    Updates 31st May 2018:

    You should see the installation wizard in step 4. Please make sure “System name” field is IP address if you only want to use IP for vCenter Server.

    Updates 5th Sep 2018:

    You may see the following error during installation.

    Could not connect to VMware Directory Service via LDAP

    It indicates vCenter Server FQDN doesn’t work. If you’re a home lab, you may want to add the DNS entries in the hosts file.

  • Troubleshooting Network Performance of Virtual Machine

    There are several layers of networking on the virtualization infrastructure. Guest operating system, Virtual Machine, ESXi driver, physical network adapters, RJ45/SFP and network switches…etc. Sometimes it’s hard to say where exactly caused a problem. Especially hardware layer problems. Today I worked on a very interesting case, it may give some ideas to troubleshooting network performance issue which is caused by hardware layers.

    A user told me he was bothered by network performance of a virtual machine. It’s slow to copy data to NFS share. But responding to “ping” command looked good. I didn’t see any issue on virtual machine layer. VMware Tools was up to date, Windows OS was patched, virtual network adapter type was VMXNET3 and VM version was also up to date.

    When I tried to copy an image file to share folder of the virtual machine, I did see sometimes speed was fast, but sometimes not. Since I have two physical uplinks, it led me to guess it could be one of the uplinks.

    After a lot of swapping and cable changing, we eventually figured out there was a bad SFP on network switch end. I was able to observe the issue by using “psping.exe” of Microsoft Sysinternals. I used the following command to send the different size of ping package to the virtual machine. Network drops were increasing when I increased package size.

    psping.exe -l <size of package> <Destination>
Example: psping.exe -l 4k xxxx.contoso.com

    The size could be 1k, 2m or even larger. I think this is a good way to identify problem outside of ESXi. Especially SFP problem as such kind of problem didn’t give any CRC or error count on network switch level.

    You can also use Windows native command “ping.exe” as following. The size unit is “bytes”. For example, you need to input 4096 if you want to send 4kb.

    ping.exe -l <size> <Destination>
Example: ping.exe -l 4096 xxx.contoso.com

     

     

  • IE 11 Window Doesn’t Change Between 4K Internal and Regular External Monitors

    Just a quick notes. If you use multiple monitors, some are 4K and some are regular resolution, you may see window display issue when move Internet Explorer between these monitors. Follow the KB below to change register to allow Internet Explorer 11 accommodates the monitor solutions.

    Internet Explorer 11 window display changes between a built-in device monitor and an external monitor

  • The older version of cis-upgrade-runner cannot be removed when upgrade vCenter Server 6.0

    When you upgrade or patch vCenter Server 6.0 for Windows, you may see following symptoms:

    “The older version of cis-upgrade-runner cannot be removed. Contact your technical support group.”

    Or error code 1063:

    “Installation of component VMware CIS upgrade runner failed with error code ‘1063’”

    That means the vCenter Server installer cannot find MSI files of existing vCenter Server services. It could be following reasons:

    • You delete MSI files in “Temp” folder of the profile you used to install vCenter Server.
    • The account you used to login and install vCenter Server was roaming profile. The profile’s “Temp” folder was automatically deleted when you reboot/logoff the server.

    vCenter Server 6.0 for Windows is consist of lot of standalone package. The upgrading process usually uninstall old packages, and then install newer packages. So the failure doesn’t impact to database or inventory data. You can re-initiate the upgrading again.

    But you cannot manually uninstall old package since upgrading process brings down vCenter services first then uninstall old packages. If you already uninstalled old packages, the upgrading process will be stuck on bring down vCenter Services stage since some processes may already be removed. For example “vmware-python” it maps to “VMware vCenter Configuration Service”. If you manually uninstalled it before launch upgrading. It removes the service. Upgrading is not able to check status of the service.

    Easiest way to get ride of this problem is

    1. Open Registry Editor (regedit) and go to the path: “HKEY_LOCAL_MACHINESOFTWAREClassesInstallerProducts”
      You would see lot of keys are there.
    2. Search keyword “vmware-“. These keys store package info of vCenter Server.
    3. Expand one of the keys. Go to “SourceList”.
    4. The value of “LastUsedSource” is path of MSI file of old vCenter Server installer.
      For example my value is “m;1;X:vcenter-serverpackages”.
    5. Make sure your server has the path mentioned in previous step (My case it’s X:vcenter-serverpackages) and old MSI files are available in the path. If it’s a CD-ROM letter, you just need mount old vCenter Server image to the drive.
    6. Copy new vCenter Server image to a local folder, uncompresse and launch installer locally.
    7. Now the upgrading process can read original packages on the mentioned path in step 4. It will automatically remove old packages by the old MSI files.

    There are two other workarounds. One is modify the value of “LastUsedSource” to reflect a new location of packages. But you still need the old MSI files be there. Another way is delete the key after you find it in step 2. (I never tested this way but it should work as it let vCenter Server installer thinks the server is brand new so installer can override the existing folders)

    I also wrote another article for upgrading error on vCenter Server 5.5 for your reference:

    CustomAction VM_InstallJRE returned actual error code 1624