Wu's Blog

MAC Address Conflict with ESXi vmkernel NIC on Cisco UCS Blades

in , , , ,

Background

I worked on a interesting case few month back. A ESXi blade was not able to bring up due to management IP address didn’t responding to ping. We tried to reconfigure IP address, re-acknowledge blade, rebuild the network, and even replaced the motherboard. It was no lucky. Eventually we figured it out that another ESXi host’s management network somehow configured same MAC address. It caused the MAC address conflict on network.

This guide will show you some tips of how to troubleshooting MAC address conflicts on ESXi and Cisco UCS level.

Some Reference

The first article you should read is “vmk0 management network MAC address is not updated when NIC card is replaced or vmkernel has duplicate MAC address”. It helps you understand why vmkernel MAC address is not updated. The solution in the KB is change MAC address manually on ESXi. Or re-create management network.

But the reality is we usually don’t know where the conflict comes from. We only know this Cisco UCS blade installed ESXi and it doesn’t responding to ping. So you may suspect it’s a hardware issue like me.

Check MAC address conflicts on Cisco UCS

There are some ways to check MAC address conflicts on Cisco UCS.

  • Login to UCS Manager by SSH and check MAC address status.
  • Export UCS Manager log and check MAC address conflicts in fwm_trace_log file.
# Login to UCS Manager
# Run following command to show mac address status.
show platform fwm info mac <mac address> <vlan id>

# Sample
show platform fwm info mac 0025.0050.11.11 141

Login to UCS Manager GUI to generate support log.

Admins -> AllFaults, Events and Audit -> Log -> TechSupport Files

Generate a ucsm log bundle. Download and extract it. There are two major files in the log bundle: UCSM_A_TechSupport.tar.gz and UCSM_B_TechSupport.tar.gz. The files correspond to their respective Fabric Interconnect.

MAC address conflicts usually occurred on one Fabric Interconnect. So you may need to check both of them. I use A side as sample. Go to extract folder -> UCSM_A_TechSupport -> sw_trace_logs -> fwm_trace_log.current

Search keyword “REGMAC seen on border port” in the log. You need to repeat same in the log of the other FI. If you can find the entries and time is recently. Then it indicate there is conflict on the MAC address outside the UCS domain.

There maybe other reasons can cause mac address issue. I wrote in Error: No NIC found with MAC address…