Monthly Archives: February 2014

Travel to Chengdu again

It’s about 5 years since last time I visited Chengdu. A beautiful city, people say “you gonna love it, and wanna live there if you come to Chengdu”. People looks like live very relax in Chengdu, they drink tea in park, play Mahjong and enjoy professional people scrape their ears (most like ears massage). I was being Chengdu for 3 month, so I’m kind familiar with this city. All memory is 5 years ago.

I was excited to get my Raspberry Pi on morning, I plan to play on it all the day. But my wife wants to discuss travel plan when we had lunch. She told me she visited Chengdu several times before, but no one is real travel, they just went to the city, got goods, then went back, she only know one place “He Hua Chi” – a clothes market. They bought clothes there and sales in their city.

Finally, we had a 3 days trip in Chengdu! That’s a crazy plan for me since I never tried planning and going in same day! We flight to Chengdu on night, and checked in a great hotel. Since it’s close to Chinese New Year, so no much people and traffic, I felt it’s like an empty city. 5 years, a lot of change, I can felt my heart beat rock when I saw some building I was familiar with. I didn’t go back to the city after I got new job in Xi’an, a lot of things I was missing every day…He’s, Minto, Chunxi road, JinLi…etc.

在出租车上,看着外边的建筑,似曾相逢,却又遥远模糊,那些熟悉的建筑还能勾起往日的点点滴滴。5年前,我还是个年轻小伙,还是个不知明天在哪里的待业青年,还是个受了就业打击的毕业不久的大学生,还是个刚刚经历了变态公司折磨的小网管,还是个创业失败的年轻人。5年后,之前的点滴成功、失败、挫折、荣耀都成为了我今天做为一个男人的生活积累、人生经验。无论是痛苦也好,高兴也好,他都是宝贵的。

成都,希望有机会再来。 Smile

Advertisements

How to decode ESXi 5.x SCSI error code

Storage is critical component for virtualization, lot of VM performance issue is related to storage latency. You may see similar error message on vmkernel log for some case:

2014-02-11T07:18:20.541Z cpu8:425351)ScsiDeviceIO: 2331: Cmd(0x4124425bc700) 0x2a, CmdSN 0xd5 from world 602789 to dev “naa.514f0c5c11a00025” failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0

It much like language of another planet when I first time saw itJ. Let’s see how to “translate” it to human language.

First, I split it to several sections:

a) 2014-02-11T07:18:20.541Z cpu8:425351)

b) ScsiDeviceIO: 2331: Cmd(0x4124425bc700) 0x2a, CmdSN 0xd5

c) from world 602789

d) to dev “naa.514f0c5c11a00025”

e) failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0

Section A shows the UTC time when the error occurred.

Section B shows what command is sent. (Actually I don’t even know what the command means is, please let me know if you know it.)

Section C shows which world the command related to.

You can found which world it is by following command

ps | grep 602789

Section D shows which storage device it show error message.

You could identify which datastore it is by following command if your datastore contains single LUN:

esxcfg-scsidevs –m naa.514f0c5c11a00025

You could also check out LUN setting and information by following command:

esxcli storage core device list –d naa.514f0c5c11a00025

esxcli storage nmp device list –d naa.514f0c5c11a00025

Section E shows SCSI sense code. That’s the part I want to give more detail.

It’s breakdown to two sections:

SCSI status codeH:0x0 D:0x2 P:0x0

H means host status

D means device status

P means plugin status

Sense data0x4 0x44 0x0

0x4 means Sense Key

0x44 means Additional Sense Code

0x0 means ASC Qualifier

Before decode, you should translate each code to NNNh notation, 0xNNN = NNNh. For example 0x7a = 7Ah, 0x77 = 77h.

SCSI status code is easy to decode. You just need to change the format and check out the code from http://www.t10.org/lists/2status.htm.

In our example H:0x0 D:0x2 P:0x0, host code 0x0 (00h) means ESX host side is good, device code 0x2 (02h) means device is not ready, plugin status code 0x0 (00h) means LUN plugin is good. (Clarify: device code 0x2 is actually means “check condition”, it’s not really means “device is not ready”, it’s just for easy understand, but looks like it confuse since “Check Condition” has different means with “Device is not Ready”. Thanks Tony point out that. )

Sense data is a little bit complicate. You have to refer two links http://www.t10.org/lists/2sensekey.htm and http://www.t10.org/lists/asc-num.txt.

In our example: 0x4 0x44 0x0, Sense Key 0x4 (4h) means HARDWARE ERROR, Additional Sense Code is 0x44 (44h) and ASC Qualifier is 0x0 (00h), combine the both code to 44h/00h, it means INTERNAL TARGET FAILURE.

Okay, then we put all decode language together:

ESX host side is good, device is not ready, LUN plugin is good because HARDWARE ERROR INTERNAL TARGET FAILURE

Actually I dumped this code from an fnic firmware/driver incompatible case. Is it make your troubleshooting more easy?J

You could also refer to following links to get more detail:

Understanding SCSI device/target NMP errors/conditions in ESX/ESXi 4.x and ESXi 5.x

Understanding SCSI host-side NMP errors/conditions in ESX 4.x and ESXi 5.x

Interpreting SCSI sense codes in VMware ESXi and ESX

Interpreting SCSI sense codes in VMware ESXi and ESX

Website comes back online!

I didn’t know ICANN request email address verification, I thought freedom everywhere outside China, but looks like it’s not. :-)

My domain was suspended due to that ICANN policy, and my QQ mailbox unable to recieve verification email from ICANN, what a unfreedom country it is! Finally I have to change my domain register mailbox to Gmail to get the email.

Shit GFW! ( Check out here to learn more about GFW )

How to Grant Multiple Domain Groups Permission to Multiple Folders on vCenter Server

If you have set of group VMs and particular group can access each set VMs, you should grant access on vSphere Client or vSphere Web Client.

SSO is slowly sometimes, you could use following CLI to do it more efficient.

New-VIPermission -Entity “Folder Name” -Principal “Domaingroup name” -Role “Role name

You could do it faster for regular folder name or group name by excel and notepad:

New-VIPermission -Entity “

Folder Name

” -Principal “

Domaingroup name

” -Role “Role name

Guess how to do it. Smile

vHBAs and other PCI devices may stop responding in ESXi 5.x when using Interrupt Remapping

Your vHBAs or other PCI devices may stop running in ESXi 5.x when using Interrupt Remapping feature.

This issue only impact to UCS blade BIOS version 1.4(3c), it has been fixed on 1.4(3j).

Please refer to http://kb.vmware.com/kb/1030265 to see how to disable Interrupt Remapping feature in ESXi 5.x

Also refer to https://tools.cisco.com/bugsearch/bug/CSCty96722.

How to Configure Serial Console for VM by Avocent ACS v6000 Virtual Advanced Console Server

Serials console is very helpful to troubleshooting Linux problem, you can see additional system message via serial console if your Linux server hung. It is essential component on physical server for troubleshooting. It’s challenge to manage serial consoles if your datacenter is very big. You may deploy console server for central management of serial consoles, you don’t have to connect your computer with serial console one by one, you just need connect console server IP follow with port name by telnet protocol.

Time comes to today, virtualization world. How you connect serial console of Linux virtual machine? Can we do exactly same like physical server? Answer is YES! There is couple of way to connect serial console of VM, each way has different benefit. I’m going to introduce the best one!

VMware has a KB article 1022303 introduces how to implement virtual console server, but it’s not very clearly, I went to wrong way by follow up the KB.

Deploy Avocent ACS v6000 virtual advanced console server

1. Download the software image from Emerson website.

2. Install the software on console server VM by follow up ACS v6000 Installer/User Guide.

Configure Linux VM serial console

1. Add a serial port to target Linux VM you want to use serial console.

2. Configure the serial port, Select Use Network option.

3. Select Client (VM initiates connection) option.

4. Input ACSID in Port URI field.

5. Select Use Virtual Serial Console Concentrator option.

6. Input telnet://console server ip:8801 in vSPC URI field.

7. Select Yield CPU on poll option.

8. Make sure Connected and Connect at power on options are selected.
Note: It indicates wrong setting on serial port if Connected option goes back to deselect status automatically after you save the setting.

Enable kernel message on Linux VM

1. Login to your target Linux VM by SSH.

2. Copy following strings to SSH, it will enable kernel message on serial console.
cat <<EOFEOF > /etc/init/serial-ttyS0.conf

# This service maintains a getty on /dev/ttyS0.

start on stopped rc RUNLEVEL=[2345]

stop on starting runlevel [016]

respawn

exec /sbin/agetty /dev/ttyS0 115200 vt100-nav

EOFEOF

3. Run following command.
initctl start serial-ttyS0

Enable serial console on Linux VM

1. Edit grub.conf by following command.
vi /boot/grub/grub.conf

2. Add following lines after hiddenmenu option.
serial –unit=1 –speed=19200

terminal –timeout=8 console serial

3. Add following line in each kernel line.
console=tty0 console=ttyS0,115200

4. Reboot VM.

Configure ASC v6000 console server

1. Login management website of ASC v6000 console server.

2. Go to PortsSerial Ports.

3. Enable ttyS1 device.

4. Go to Access option, you will see the serial console is automatically mapped to serial console of target Linux VM.

Validation

1. Login serial console of target Linux VM via console server by telnet, SSH or serial viewer.

2. Login SSH of target Linux VM directly.

3. On SSH session, run following command to trigger kernel message.
echo h > /proc/sysrq-trigger

4. You will see message on serial console screen.