Wu's Blog

My IT Adventure

Category: English

English version of my posts.

  • CVE-2017-5754, CVE-2017-5753 and CVE-2017-5715 (Spectre and Meltdown)

    You may know there are 3 vulnerabilities recently noticed by industry. Long story to short, kernel address space exposed to hackers when processors running user space code. It’s not only impact to Intel processors but also AMD and ARM. CVE-2017-5715 is a hardware issues that only apply certain firmware can fix the vulnerabilities. CVE-2017-5754 and CVE-2017-5753 need to apply OS patches to change how codes access kernel address space. Following are some useful links just for your reference.

    CVE-2017-5753

    CVE-2017-5715

    CVE-2017-5754

    VMware: https://www.vmware.com/security/advisories/VMSA-2018-0002.html (For CVE-2017-5753 and CVE-2017-5715. VMware has not published anything for CVE-2017-5754 yet.)

    Microsoft: https://support.microsoft.com/en-gb/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution
    https://support.microsoft.com/en-gb/help/4073119/protect-against-speculative-execution-side-channel-vulnerabilities-in

    HPE: http://h22208.www2.hpe.com/eginfolib/securityalerts/SCAM/Side_Channel_Analysis_Method.html

    Cisco: https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180104-cpusidechannel

  • VMware Remote Console Freeze or Black Screen

    The latest version of VMware Remote Console is 10.0.2. There should be some functions changing in the release. You may see following symptoms after upgrading to the version.

    1. The virtual machine console screen is black.
    2. Console screen works properly if you just hit buttons on VMware Remote Console window.
    3. The console screen freeze once VMware Remote Console grabs mouse or keyboard in screen.

    The cause is your  anti-virus software blocked the internal functions of grab mouse/keyboard behavior of VMware Remote Console. Try disable anti-virus temporarily.

  • How To Migrate Parent Disk on Hyper-V 2012

    If you are using Microsoft Hyper-V 2012 and “Differencing Disk” you may get trouble when you want to move whole VMs to another location due to “Parent Disk” migration is not so easy. Following is the steps to move parent disk on Hyper-V server.

    Preparation

    I assume you want to move bunch of virtual machines. First of all you need to get disk list of virtual machines. Following is a script to grab all parent and differencing disks on a Hyper-V server.

    $VMs = Get-VM 
Foreach ($VM in $VMs)
{
  $VHDs = Get-VHD -Path $VM.harddrives.path
  foreach ($VHD in $VHDs)
  {
     [pscustomobject]@{
         Name = $VM.name
         VHDType = $VHD.VhdType
         VHD = $VHD.Path
         ParentVHD = $VHD.ParentPath
     }
  }
}

    Save it to “Get-vhdParent.ps1”. Launch PowerShell by administrator right. Run following command to get parent disk table.

    .Get-vhdParent.ps1 | format-table -autosize

    Now you have disk list in hand.

    Move parent disks to new location

    Parent disk moving is simple. Just copy the parent disk to new location. I suggest make multiple copies if you have large number of virtual machines linked to a parent disk. The reason is if the parent disk failed, at lease it’s not impact to all linked virtual machines. You can also distribute the duplicated parent disks to multiple location to avoid single location failure.

    Re-configure parent disks for virtual machine

    To be safe, I suggest grab parent disk information again by following command:

    Get-VHD -Path VHDPath

    Replace “VHDPath” with real differencing disk path of the virtual machine.

    The output shows what’s the linked parent disk. Then run the command below to reconfigure parent disk to new location.

    Set-VHD -Path VHDPath -ParentPath ParentVHDPath

    You should get nothing return if it’s successfully.

    If you manage Hyper-V virtual machines by System Center Virtual Machine Manager. The new parent disk is reflected after you right click the virtual machine and do a “Refresh” in System Center Virtual Machine Manager console.

     

     

     

  • Cannot Complete File Creation Operation When Storage vMotion

    Just quick notes. I saw following error  when do storage vMotion.

    Cannot Complete File Creation Operation.

    When check /var/log/hostd.log. I saw following errors:

    2017-11-28T02:51:04.476Z info hostd[76A80B70] [Originator@6876 sub=Vimsvc.TaskManager opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Task Created : haTask--vim.host.OperationCleanup
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] CopyFromEntry: Hostlog_Dump: Hostlog /vmfs/volumes/598700ee-ec
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] UUID: 28dbb1b5-a9d8-e311-1061-03300000002d
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] MigID: 1511837464286041
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] HLState: none
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] ToFrom: none
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] MigType: invalid
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] OpType: nfc
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] WorldID: 0
2017-11-28T02:51:04.478Z warning hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Hostlog_Flush: Failed to open hostlog /vmfs/volumes/598700e
2017-11-28T02:51:04.478Z warning hostd[772C2B70] [Originator@6876 sub=Vcsvc.OCM opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] PersistToDisk: failed to persist entry /vmfs/volumes/5
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Default opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] AdapterServer caught exception: vim.fault.CannotCreateFile
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Vimsvc.TaskManager opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Task Completed : haTask--vim.host.OperationClean
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Activation [N5Vmomi10ActivationE:0x75395c80] : Invoke do
2017-11-28T02:51:04.478Z verbose hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Arg entry:
--> (vim.host.OperationCleanupManager.OperationEntry) {
--> hlogFile = "/vmfs/volumes/598700ee-ec0f9918-5b56-000000000000/XXX-VM-01/XXX-VM-01-375f29ae.hlog",
--> opId = 1511837464286041,
--> opState = "running",
--> opActivity = "nfc",
--> curHostUuid = "28dbb1b5-a9d8-e311-1061-03300000002d",
--> }
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Throw vim.fault.CannotCreateFile
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Result:
--> (vim.fault.CannotCreateFile) {
--> faultCause = (vmodl.MethodFault) null,
--> file = "/vmfs/volumes/598700ee-ec0f9918-5b56-000000000000/XXX-VM-01/XXX-VM-01-375f29ae.hlog",
--> msg = ""
--> }

    It indicates there is a file cannot be created during migration. Further check on VM configuration file (.vmx) I noticed following parameter existing but the file doesn’t existing.

    migrate.hostlog = "XXX-VM-01-375f29ae.hlog"

    You cannot create the file directly. Workaround is create a .hlog file with other name then rename it to the same name.

    BTW, there is a bug on ESXi 6.0 U1 for similar issue, but I saw this problem  on  U2. Just for your reference below.

    Storage migration of a virtual machine with a name beginning with core fails with the error: Relocate virtual machine coreXX Cannot complete the operation because the file or folder coreXX-XXXXX.hlog already exists

  • Virtual Machine Cloning Is Failed At 33%

    I got two exactly same hardware, installed same ESXi version. Somehow cloning from  other ESXi to one server was working, but another one always failed at 33%.

    It only impacts to existing VM cloning but not impact to new created virtual  machines. I spent lot time on the troubleshooting. We replaced cables, switch ports, reinstalled ESXi.

    There was no  abnormal logs except following:

    2017-11-15T05:47:36.023Z [FFF001A0 verbose 'NfcManager' opID=F39DF7E6-00002211-da-c-bf] [NfcClient] Closing NFC connection to server

2017-11-15T05:47:36.023Z [FFF001A0 warning 'Libs' opID=F39DF7E6-00002211-da-c-bf] SSL: Unknown SSL Error

2017-11-15T05:47:36.023Z [FFF001A0 info 'Libs' opID=F39DF7E6-00002211-da-c-bf] SSL Error: error:1409E10F:SSL routines:SSL3_WRITE_BYTES:bad length

    VMware support eventually provided me following KB to workaround the problem. Looks like it’s a bug on ESXi 5.5 2068190.

    Disabling SSL for NFC data traffic in vCenter Server

    Cloning or deploying from a template takes longer time after upgrading to VMware vSphere 5.1 Update 2 and 5.5

  • How To Get Used Space By PowerShell

    I searched internet but hard to find an easy way to get used space on Windows Server.

    Following is two lines PowerShell command to get used space on  Windows 2012 R2 Server.

    Get-WmiObject win32_logicaldisk | select deviceid,@{n="Size";e={[math]::Round(($_.size/1GB),2)}},@{n="Used Space";e={[math]::Round((($_.Size-$_.FreeSpace)/1GB),2)}}

  • Offline Installer of IT Pro Tools

    Following are offline or portable executives of some IT tools. I’ll continue update the list whenever I find a new one.

    Google Chrome offline installer

    Flash Player offline installer for Chrome

    Flash Player offline install for FireFox

    Flash Player offline update for IE 11 on Windows 10 (Fix IE crash problem when login to vSphere Web Client)

  • Could Not Complete Network Copy For File During VM Cloning

    This error may only appears on legacy ESXi hosts. The cloning of virtual machine throws error at 33% of the task.

     Clone virtual machine
Could not complete network copy for file 
/vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST/TEST.vmdk
Copying Virtual Machine files

    You may see following logs on vpxa.log of source ESXi host.

    2017-11-02T02:51:20.238Z [FFF43B70 info 'Libs' opID=812BF517-00000667-f0-34-64] SSL: syscall error 32: Broken pipe
2017-11-02T02:51:20.238Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2017-11-02T02:51:20.238Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcFile_SendMessage: data send failed:
2017-11-02T02:51:20.238Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NFC_NETWORK_ERROR
2017-11-02T02:51:20.239Z [FFF43B70 error 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcClient] File transfer [/vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST/TEST.vmdk -> /vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST1/TEST1.vmdk] failed: The operation experienced a network error
2017-11-02T02:51:20.239Z [FFF43B70 verbose 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcClient] Closing NFC connection to server
2017-11-02T02:51:20.239Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] SSL: Unknown SSL Error
2017-11-02T02:51:20.239Z [FFF43B70 info 'Libs' opID=812BF517-00000667-f0-34-64] SSL Error: error:1409E10F:SSL routines:SSL3_WRITE_BYTES:bad length
2017-11-02T02:51:20.239Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2017-11-02T02:51:20.239Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR
2017-11-02T02:51:20.239Z [FFF43B70 error 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcWorker] Error encountered while processing copy spec for file [ds:///vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST/TEST.vmdk -> ds:///vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST1/TEST1.vmdk]:
--> vim.fault.NetworkCopyFault
2017-11-02T02:51:20.239Z [FFF43B70 error 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcManagerImpl] Copy operation failed with error: vim.fault.NetworkCopyFault

    You may see following logs in vpxa.log of destination ESXi host.

    2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] SSL: Unknown SSL Error
2017-11-02T02:51:18.289Z [304EEB70 info 'Libs' opID=task-internal-2164-739c5f01] SSL Error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcNetTcpRead: bRead: -1
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcNet_Recv: requested 262144, recevied only 16384 bytes
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcFile_RecvMessage: data recv failed. retval = 3, expected 262144
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcFile_ContinueReceive: failed to Recv message
2017-11-02T02:51:18.446Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcProcessStreamMsg: failed to receive file data
2017-11-02T02:51:18.446Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcServerLoop: NfcServer_HandleRead returned an error : NFC_NETWORK_ERROR
2017-11-02T02:51:18.446Z [304EEB70 error 'provisioningvpxNfcServer' opID=task-internal-2164-739c5f01] [VPXNFCSERVER] Nfc server failed with return value : NFC_NETWORK_ERROR
2017-11-02T02:51:18.446Z [304EEB70 verbose 'provisioningvpxNfcServer' opID=task-internal-2164-739c5f01] [VPXNFCSERVER] Closing NFC session

    It indicates the VM may be created on an older ESXi host or VMware Workstation. Somehow it imported to current ESXi host. Solution is create a new VM on the ESXi host and attach only virtual disks of the problematic VM.

  • Disable PXE Boot for Individual vNIC on Virtual Machine

    To achieve Auto Deploy I’d like to control PXE boot process. I want the vNICs of management network can do PXE boot only. That’s because DHCP server may learns incorrect MAC address of management network if ESXi host boots up by non-management network NICs.

    The psychical servers are easy to disable PXE boot feature of individual network adapters in BIOS or server profile. Virtual machines is tricky. Following is how to do it. It’s useful for home lab.

    1. Make sure your ESXi VM uses E1000E vNIC. You can only disable PXE boot for all vNICs in one time if type is vmxnet3. And nested ESXi doesn’t support E1000 vNICs.
    2. Go to the VM folder and edit vmx file.
    3. You should see similar entries below. The vNIC name starts from  ethernet0 to ethernetn. It matches vmnic0 to vmnicx on ESXi.

      ethernet1.virtualDev = “e1000e”

    4. I have 4 vNICs. I want to keep PXE boot for ethernet0 and ethernet1. So I only disable it on rest of vNICs. Add following lines to vmx file.

      ethernet2.opromsize = “0”
      ethernet3.opromsize = “0”

    5. Save the vmx file and quit.
    6. Power on the VM.

    Please make sure you power on the virtual machines by vSphere Client or vSphere Web Client. As you may know VM console may opened by VMware Workstation if workstation and vSphere Client both existing on your computer. Looks like sometimes the parameter doesn’t work if you power on the  VM by VMware Workstation.

    VMware KB Disabling Network boot option from appearing in a virtual machine’s BIOS (1014906) talks about same thing but the value of parameters looks like incorrect for ESXi 6.5.

  • How To Find Non-tagged ESXi Hosts

    There are plenty of scripts to find tagged ESXi hosts. But what if you want to find out all ESXi hosts not be tagged? Following is a simple script:

    Compare-Object ((Get-VMHost | Get-TagAssignment).Entity | select -uniq) (Get-VMHost)

    The output is similar like following:

    InputObject      SideIndicator
-----------      -------------
esx1         =>  
esx2         =>  
esx3         =>

    The => indicates ESXi hosts in InputObject are not tagged.

    If return is nothing, it means all ESXi hosts are tagged.

    Please refer to “Using the Compare-Object Cmdlet” for detail.