Cisco UCS Error: Applying moref properties. Remote-Invocation-Error.

Cisco UCS Manager shows following errors with code “F1001201”.

Applying moref properties(FSM:sam:dme:MorefImportRootApplyMoRefs)​. Remote-Invocation-Error: FSM Retries Exhausted.

The reason is Cisco UCS Manager cannot imports some configurations when the UCS is setup.

Solution:

  1. Login to Cisco UCS Manager by SSH.
  2. Run following commands:
    • # scope system
  3. If you want to see what is pending on importing please run following commands, otherwise skip this step:
    • # show pending-import
    • # show pending-import fsm status expand detail
  4. Delete pending imports:
    • # delete pending-import
    • # commit-buffer
  5. The alerts should go away now in Cisco UCS Manager GUI.

This procedure is online operation without downtime required.

Agentless Management with PowerShell for DELL iDRAC

DELL provides agentless management of iDRAC by PowerShell. It’s more convenience for enterprise level infrastructure compare with HPE Scripting Tools or Cisco PowerTools. You don’t have to install any extra software to retrieve hardware data from iDRAC

DELL provides an article “Agentless Management with PowerShell 3.0 – CIM Cmdlets and iDRAC/Lifecycle Controller” with a sample code which is very detail. The scripts leverage public repos on internet: schemas.dell.com.

But what if your infrastructure is offline or internet is blocked?

After few hours dig into it. I found a workaround works in my environment.

  1. Go to DCIM library profile to find profile of what your iDRAC. (I use DCIM System Info Profile for example)
  2. The document tells what’s the ” Implementation Namespace” and “Central Class”.
  3. Follow sample code in DELL document but when you connect CIM session use following format:
Get-CimInstance -CimSession $session -ClassName <class name> -Namespace <Name space name>

In my sample, the command is:

Get-CimInstance -CimSession $session -ClassName DCIM_SystemView -Namespace "root/dcim"

I have some posts talk about PowerShell. Here is the article to get space utilization of Windows Servers.

Cannot Open Cisco UCS KVM Console By Java

When you lunch KVM console in Cisco UCS Manager. You probably get following error message:

Unable to launch the application

Error: you can not run this program because your system deployment.config file states that an enterprise configuration file is mandatory…

This is caused by Java. There are two things you can try to fix KVM console:

  • Install Java on a directory without “space”. For example, install it on C:\java\jre7.
  • Delete Sun folder in C:\windows\. But please make a backup of the folder since it may contains some special configuration of your enterprise.

I have another blog talking about UCS KVM issue: Cisco UCS Blade Cannot Get IP Address for KVM

Emulex OneConnect OCe10102 on ESXi 6.0

Please refer to following post for basic troubleshooting of Emulex OneConnect.

How to Install Proper Drivers for 3rd Party Network Adapter on ESXi 5.x

I have a box uses Emulex OneConnect OCe10102 network adapters. The adapter is quite old and Emulex brand card doesn’t support ESXi 6.0. I upgraded the server to ESXi 6.0 and the Emulex adapters lost.

In the initial troubleshooting, I noticed that the adapters are still visible in BIOS. So it should be some driver level issues. I checked VMware Compatibility Guide. The model OCe10102 doesn’t support by ESXi 6.0.

If you run the following command you will still be able to see the adapters in PCI list on ESXi.

[code language="perl"]
esxcli hardware pci list
[/code]

So it indicates the adapters are not visible in ESXi since the newer Emulex driver doesn’t contain the model of the adapter in ESXi 6.0 native driver.

Then I uninstalled the native Emulex driver for ESXi 6.0 by the following command and rebooted the ESXi host.

[code language="perl"]
esxcli software vib remove -n elxnet
[/code]

The adapters still not visible after rebooting since no any drivers for Emulex adapters. Then I downloaded the Emulex drivers for ESXi 5.5 on VMware website and uploaded the “offline” package in the zip file to /tmp directory of the host. Then installed the driver by the following command:

[code language="perl"]
esxcli software vib install -d "/tmp/xxxxx.zip"
[/code]

The adapters appeared after rebooting the host.

“The Update is Not Applicable to Your Computer” When Install KB3046101

  HPE 3PAR upgrading team usually sends a per-requisites before upgrading. One thing in the guide incorrect is the Windows 2012 required patch KB3046101.     You may see error below when you install the path on Windows 2012 server.    
The update is not applicable to your computer
      The reason is the version of mpio.sys and msdsm.sys on your server is higher than 6.3.9600.17809. Usually, because the server installed KB3121261 already. You can ignore KB3046101.

Vlan ‘xxx’ resolved to unsupported VLAN ID in Cisco UCS Manager

You may need only 1 IP address for blade console in Cisco UCS Manager. You can follow Understanding “Management IP” of Cisco UCS Manager to configure it. You may see warning “Vlan ‘xxx’ resolved to unsupported VLAN ID” when you delete existing inbound and outbound IP pools if you are trying to clean up existing management IP pools.

That’s because inbound IP address for blade is not cleaned. You have to go to “Equipment” -> “Chassis” -> Target chassis -> “Servers” -> Target server -> Go to “Inventory” tab -> “CIMC” tab -> Click “Change Inbound Management IP” -> Remove existing VLAN and IP pool.

You will see inband IP tab is blank once it’s saved. Please note, the IP address reassign back after 1 minute if you clicked “Delete Inband Configuration” instead of “Change Inbound Managemnt IP“.

Understanding “Management IP” of Cisco UCS Manager

IP address for KVM in Cisco UCS Manager is different with HPE servers. It may assign multiple IP addresses to same blade if you don’t configure it properly. In my case each blade gets 3 IP addresses!

There are actually 3 types of IP address for KVM. (Cisco manual says 2):

  • Outbound Management IPs.
  • Inbound Management IPs for Blades.
  • Inbound Management IPs for Service Profiles.

Outbound Management IP” is default for KVM. Every new deployed blade will try to get a DHCP IP address over Cisco UCS Fabric Interconnect management port in same VLAN.

The confusion is the 2nd and 3rd IPs.  “Inbound Management IPs for Blades” is from “hardware” perspective. “Inbound Management IPs for Service Profiles” is from “logical” perspective.

If you go to Equipment -> Chassis -> blade -> Click the KVM to go console. You get console over either Outbound Management IP or Inbound Management IPs for Blades.

If you go to Servers -> Service Profiles -> Click the KVM of a service profile. You get console over either Outbound Management IP or Inbound Management IPs for Service Profiles.

If you want to configure just 1 IP for a blade whatever it’s for hardware or service profile. You need to do following:

  1. Delete the range of the default ext-mgmt in IP Pools of LAN node in Cisco UCS Manager.
  2. Create a new inbound IP pool and a VLAN group without uplink.
  3. Choose the VLAN and inbound IP pool in LAN Cloud -> Global Policies -> Inbound Profile.
  4. Assign the VLAN and inbound IP pool to templates or service profile.

Refer to Setting the Management IP Address of Cisco UCS Manager manual for detail.

BTW, you may see Vlan ‘xxx’ resolved to unsupported VLAN ID in Cisco UCS Manager when you clean up existing IP pool and create new inbound pool.

“x/xx on FI-A is connected by a unknown server device” on Cisco UCS

You may see following errors in ‘info’ category of error messages in the Cisco UCS Manager after upgrading infrastructure firmware to 3.2.x.

“x/xx on FI-A is connected by a unknown server device”

This is bug documented in CSCvk76095. You have to reset the port on FI to fix it.

  1. Go to “Equipment” in Cisco UCS Manager.
  2. Go to “Fabric Interconnects” -> Go to the corresponding FI.
  3. Right-click the port x/xx -> Choose “Disable“.
  4. You will see multiple major faults. Wait for 5 seconds.
  5. Right click the port x/xx -> Choose “Enable“.
  6. All warnings disappeared after 5 mins. You may still see the warning in GUI due to cache. Relogin and check.

This change impacts to one link between IOM and the FI port. You need downtime if the IOM only has a single path. I don’t see any impact to ESXi blades in the pod.

Connect to New Provisioned Raspberry Pi Less than $3

The IP configuration of new provisoined Raspberry Pi struggled me a long time. I need to connect to a monitor so I login to system and configure IP address. The problem was I don’t have monitor. I only have a laptop.

Last year, my old laptop dead. I connected the laptop monitor to a HDMI board to my Raspberry Pi. It’s not a low cost solution, it costed me more than $10. And the monitor, cables and board looks uglily.

IMG_1844

Actually there is another solution to leveraging laptop keyboard and monitor. It’s serials port to console. Something similar like when you configure Cisco network switches. Following is how to do it. I achieve that on Raspberry Pi 2.

  1. You need to buy a USB to TTL device with chipset CP2102.
  2. Connect the pins to Raspberry Pi 2. Refer here for GPIO layout.
    TXD > Pi RXD Pin #10 (GPIO 16)
    RXD > Pi TXD Pin #08 (GPIO 15)
    GND > Pi GND Pin #6
  3. Connect the USB to laptop. You will see a device in ‘Device Manager’ needs drivers.
  4. Download driver and install.
  5. Download Putty and install.
  6. Open Putty and “Serial”.
  7. “Serial line” is COM3 or COM4.
  8. “Speed” is 115200.

The USB to TTL I bought on Taobao (Chinese version of Aliexpress). It’s around $1.2 including shipping.

Show CDP Neighbor of Cisco UCS Uplinks

There are two ways to know which network switch ports the network uplinks of Cisco UCS Fabric Interconnects are connected to.

By CLI

  • SSH to the Cisco UCS Manager.
  • Connect to FI-A.
# connect nxos a
  • Show neighbor of network uplinks.
# show cdp neighbor interface ethernet <port num>

By PowerShell

  • Make sure Cisco PowerTool (For UCS Manager) is installed.
  • Enabling the Information Policy via UCSM GUI.
    • Go to “Equipment” -> “Policies” tab -> “Global Policies” tab -> “Info Policy” area.
    • Change to “Enabled“. (No impact to running blades)
  • Open a PowerShell window.
  • Connect to the UCS Manager.
# Connect-Ucs <UCS FQDN>
  • Show CDP neighbor details.
# Get-UcsNetworkLanNeighborEntry

Side notes

Following command can shows network switch name, network switch ports and FI ports

# Get-UcsNetworkLanNeighborEntry | Select deviceid,remoteinterface,localinterface

If you prefer to enable the “Info Policy” by PowerShell, run following command

# Get-UcsTopInfoPolicy | Set-UcsTopInfoPolicy -State enabled -Force