Wu's Blog

My IT Adventure

Category: English

English version of my posts.

  • Error 2931 The connection to the VMM agent on the virtualization server was lost

    Windows Server 2012, the biggest competitor of VMware vSphere. There are adequate reason to use Hyper-V 2012 instead of vSphere 5.x, but it’s still very hard to for newbie, we spend more than 30 hours to try figure out how to create cluster on SCVMM 2012 SP1, the software is easy to install, but hard to configure. I saw “failed” everywhere, it’s not a mature product in my view.

    We installed Windows Server 2012 data center edition on HP BL460, storage is NetApp FAS2240 (Maybe wrong, I’m not storage guy). We got following error message when we created Hyper-V Cluster on SCVMM2012 SP1.

    Error (2931)
    VMM is unable to complete the request. The connection to the VMM agent on the virtualization server (xxx) was lost.
    Unknown error (0x80338029)

    Recommended Action
    Ensure that the Windows Remote Management (WS-Management) service and the VMM agent are installed and running and that a firewall is not blocking HTTPS traffic.

    This can also happen due to DNS issues. Try and see if the server (dcahyv04.amat.com) is reachable over the network and can be looked up in DNS. You can ping the virtualization server from VMM management server and make sure that the IP address returned matches the IP address locally obtained from the virtualization server.

    If the error still persists, restart the virtualization server, and then try the operation again.

    SCVMM job failed on Mounts storage disk on xxxx.

    Initially I thought that’s something wrong with services, I checked the mentioned Windows Remote Management service, but it’s up and running. Then I noticed WINS servers was not set, but still no lucky.

    Why the job always failed on mount storage? Maybe something related to disk operation? SCVMM server is remote server, it must be operates disk remotely, so I tried connect Hyper-V server by Computer Management tool remotely, it show me RPC is unavailable when I click Disk Management node. Aha…firewall problem, that’s because SCVMM server disabled firewall, but Hyper-V server enabled, the RPC ports was blocked by client side.

    Sometimes cluster creating can be successful after I disabled firewall, but still Hyper-V server looks like hard to mount storage.

    Since SCVMM mount/unmount storage on each Hyper-V hosts during cluster creating, it takes very long time to mount storage before the job failed, we suspected that’s something related to storage, finally, we installed NetApp Host Utilities 6.0.1 and NTAP MPIO 4.0.1 to solved the problem.

    To summarize, you must enable remote management for Hyper-V host, such as Remote Register service…etc, allows required ports in Windows firewall and storage MPIO plugin should be installed as well. BTW, you should disable UAC on Windows Server 2012, it’s different with Windows Server 2008, check http://social.technet.microsoft.com/wiki/contents/articles/13953.windows-server-2012-deactivating-uac.aspx.

    That’s just first step to make Hyper-V successful. 🙂

  • How to Add VMware PowerCLI to Standard PowerShell Enviroenment

    1. create a file with name “Profile.ps1” under %windir%system32WindowsPowerShellv1.0profile.ps1

    2. Add following content to the file.

    # Adds the base cmdlets
    Add-PSSnapin VMware.VimAutomation.Core
    # Add the following if you want to do things with Update Manager
    #Add-PSSnapin VMware.VumAutomation
    # This script adds some helper functions and sets the appearance. You can pick and choose parts of this file for a fully custom appearance.
    . “C:Program FilesVMwareInfrastructurevSphere PowerCLIScriptsInitialize-PowerCLIEnvironment.ps1”

    You need administrator permission to create file on the system32 location.

  • How to configure vSAN on nested ESXi hosts with SSD hard disk

    There are lot of articles introduce vSAN feature and steps by steps guide. I referred William Lam’s article & Duncan’s article to configure vSAN on my lab, I was true I exactly followed his steps to configure the vSAN, but I can not saw anything under disk field under Disk Management.

    Please note: Following steps does not work for ESXi 6.0 RC on VMware Workstation 10. You have to set scsix:y.virtualssd = 0 in vmx file to mark the disk as non-SSD. Please refer to William’s article for detail.

    After looked into it deeper, I found something interesting:

    esxcli storage core device list

    I got that output:

    mpx.vmhba1:C0:T1:L0
    Display Name: Local VMware, Disk (mpx.vmhba1:C0:T1:L0)
    Has Settable Display Name: false
    Size: 5120
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0
    Vendor: VMware,
    Model: VMware Virtual S
    Revision: 1.0
    SCSI Level: 2
    Is Pseudo: false
    Status: on
    Is RDM Capable: false
    Is Local: true
    Is Removable: false
    Is SSD: true
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: unsupported
    Other UIDs: vml.0000000000766d686261313a313a30
    Is Local SAS Device: false
    Is Boot USB Device: false
    No of outstanding IOs with competing worlds: 32

    Initially, I thought that disk marked as SSD since I ran command to enable SSD. Actually it’s not like that, it shows SSD since my hard disk is SSD!!!! I don’t have to run the command introduced in the articles to turn SSD on, it’s nature SSD. lol

    What I need to do is actually totally oppositely. That’s the steps I used to enable vSAN:

    1. Create two disks.

    2. Login ESXi hosts by SSH.

    3. Run following command, find out the two disks you want to use for vSAN. Record runtime name.

    esxcli storage core device list

    4. Run following command to disable SSD for one disk.

    esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device vmhba1:C0:T2:L0 –option “disable_ssd”

    5. Follow up the articles above to enable vSAN ports, create clusters, enable vSAN on cluster and join ESXi hosts to clusters.

  • How to setup NTP services by PowerCLI

    NTP service is very important for troubleshooting, vmkernel log timestamp is incorrect if your NTP service is not running and ESXi system time is wrong. It can also impact to VM system time even you disable time synchronization on VMware Tools since VM still need to sync time with ESXi after awake from suspended status, finish vMotion, or revert from snapshot.

    I know it’s simple to configure NTP services on single how, what if you want to configure NTP service on massed hosts?

    Basically we have 3 steps to make sure NTP service working properly:

    • Configure NTP server IP address.
    • Bring up NTP service.
    • Set services startup along with ESXi system.

    Let’s try PowerCLI:

    Get-VMHOST -Location Cluster Name | Add-VMHostNtpServer -NtpServer “NTP server address“

    Get-VMHOST -Location Cluster Name | Get-VMHostService| Where-Object {$_.key -eq “ntpd”} | Start-VMHostService

    Get-VMHOST -Location Cluster Name | Get-VMHostService| Where-Object {$_.key -eq “ntpd”} | Set-VMHostService –Policy On

  • Travel to Chengdu again

    It’s about 5 years since last time I visited Chengdu. A beautiful city, people say “you gonna love it, and wanna live there if you come to Chengdu”. People looks like live very relax in Chengdu, they drink tea in park, play Mahjong and enjoy professional people scrape their ears (most like ears massage). I was being Chengdu for 3 month, so I’m kind familiar with this city. All memory is 5 years ago.

    I was excited to get my Raspberry Pi on morning, I plan to play on it all the day. But my wife wants to discuss travel plan when we had lunch. She told me she visited Chengdu several times before, but no one is real travel, they just went to the city, got goods, then went back, she only know one place “He Hua Chi” – a clothes market. They bought clothes there and sales in their city.

    Finally, we had a 3 days trip in Chengdu! That’s a crazy plan for me since I never tried planning and going in same day! We flight to Chengdu on night, and checked in a great hotel. Since it’s close to Chinese New Year, so no much people and traffic, I felt it’s like an empty city. 5 years, a lot of change, I can felt my heart beat rock when I saw some building I was familiar with. I didn’t go back to the city after I got new job in Xi’an, a lot of things I was missing every day…He’s, Minto, Chunxi road, JinLi…etc.

    在出租车上，看着外边的建筑，似曾相逢，却又遥远模糊，那些熟悉的建筑还能勾起往日的点点滴滴。5年前，我还是个年轻小伙，还是个不知明天在哪里的待业青年，还是个受了就业打击的毕业不久的大学生，还是个刚刚经历了变态公司折磨的小网管，还是个创业失败的年轻人。5年后，之前的点滴成功、失败、挫折、荣耀都成为了我今天做为一个男人的生活积累、人生经验。无论是痛苦也好，高兴也好，他都是宝贵的。

    成都，希望有机会再来。 Smile

  • How to decode ESXi 5.x SCSI error code

    Storage is critical component for virtualization, lot of VM performance issue is related to storage latency. You may see similar error message on vmkernel log for some case:

    2014-02-11T07:18:20.541Z cpu8:425351)ScsiDeviceIO: 2331: Cmd(0x4124425bc700) 0x2a, CmdSN 0xd5 from world 602789 to dev “naa.514f0c5c11a00025” failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0

    It much like language of another planet when I first time saw itJ. Let’s see how to “translate” it to human language.

    First, I split it to several sections:

    a) 2014-02-11T07:18:20.541Z cpu8:425351)

    b) ScsiDeviceIO: 2331: Cmd(0x4124425bc700) 0x2a, CmdSN 0xd5

    c) from world 602789

    d) to dev “naa.514f0c5c11a00025”

    e) failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0

    Section A shows the UTC time when the error occurred.

    Section B shows what command is sent. (Actually I don’t even know what the command means is, please let me know if you know it.)

    Section C shows which world the command related to.

    You can found which world it is by following command

    ps | grep 602789

    Section D shows which storage device it show error message.

    You could identify which datastore it is by following command if your datastore contains single LUN:

    esxcfg-scsidevs –m naa.514f0c5c11a00025

    You could also check out LUN setting and information by following command:

    esxcli storage core device list –d naa.514f0c5c11a00025

    esxcli storage nmp device list –d naa.514f0c5c11a00025

    Section E shows SCSI sense code. That’s the part I want to give more detail.

    It’s breakdown to two sections:

    SCSI status code – H:0x0 D:0x2 P:0x0

    H means host status

    D means device status

    P means plugin status

    Sense data – 0x4 0x44 0x0

    0x4 means Sense Key

    0x44 means Additional Sense Code

    0x0 means ASC Qualifier

    Before decode, you should translate each code to NNNh notation, 0xNNN = NNNh. For example 0x7a = 7Ah, 0x77 = 77h.

    SCSI status code is easy to decode. You just need to change the format and check out the code from http://www.t10.org/lists/2status.htm.

    In our example H:0x0 D:0x2 P:0x0, host code 0x0 (00h) means ESX host side is good, device code 0x2 (02h) means device is not ready, plugin status code 0x0 (00h) means LUN plugin is good. (Clarify: device code 0x2 is actually means “check condition”, it’s not really means “device is not ready”, it’s just for easy understand, but looks like it confuse since “Check Condition” has different means with “Device is not Ready”. Thanks Tony point out that. )

    Sense data is a little bit complicate. You have to refer two links http://www.t10.org/lists/2sensekey.htm and http://www.t10.org/lists/asc-num.txt.

    In our example: 0x4 0x44 0x0, Sense Key 0x4 (4h) means HARDWARE ERROR, Additional Sense Code is 0x44 (44h) and ASC Qualifier is 0x0 (00h), combine the both code to 44h/00h, it means INTERNAL TARGET FAILURE.

    Okay, then we put all decode language together:

    ESX host side is good, device is not ready, LUN plugin is good because HARDWARE ERROR INTERNAL TARGET FAILURE

    Actually I dumped this code from an fnic firmware/driver incompatible case. Is it make your troubleshooting more easy?J

    You could also refer to following links to get more detail:

    Understanding SCSI device/target NMP errors/conditions in ESX/ESXi 4.x and ESXi 5.x

    Understanding SCSI host-side NMP errors/conditions in ESX 4.x and ESXi 5.x

    Interpreting SCSI sense codes in VMware ESXi and ESX

    Interpreting SCSI sense codes in VMware ESXi and ESX

  • Just another blog

    Hi there,

    I built a new personal website, hence I’ll stop update here.

    Please go my Wu’s Blog to get latest articles.

  • Website comes back online!

    I didn’t know ICANN request email address verification, I thought freedom everywhere outside China, but looks like it’s not. 🙂

    My domain was suspended due to that ICANN policy, and my QQ mailbox unable to recieve verification email from ICANN, what a unfreedom country it is! Finally I have to change my domain register mailbox to Gmail to get the email.

    Shit GFW! ( Check out here to learn more about GFW )

  • How to Grant Multiple Domain Groups Permission to Multiple Folders on vCenter Server

    If you have set of group VMs and particular group can access each set VMs, you should grant access on vSphere Client or vSphere Web Client.

    SSO is slowly sometimes, you could use following CLI to do it more efficient.

    New-VIPermission -Entity “Folder Name” -Principal “Domaingroup name” -Role “Role name”

    You could do it faster for regular folder name or group name by excel and notepad:

    New-VIPermission -Entity “

    Folder Name

    ” -Principal “

    Domaingroup name

    ” -Role “Role name“

    Guess how to do it. Smile

  • vHBAs and other PCI devices may stop responding in ESXi 5.x when using Interrupt Remapping

    Your vHBAs or other PCI devices may stop running in ESXi 5.x when using Interrupt Remapping feature.

    This issue only impact to UCS blade BIOS version 1.4(3c), it has been fixed on 1.4(3j).

    Please refer to http://kb.vmware.com/kb/1030265 to see how to disable Interrupt Remapping feature in ESXi 5.x

    Also refer to https://tools.cisco.com/bugsearch/bug/CSCty96722.