HS22V ESXi 5.5 Upgrade 10Gb Network Problem

This post is applicable to customers using IBM BladeCenter HS22 and HS22V blades with the Emulex 10Gb adapter cards that connect to the IBM 10Gb Virtual Fabric Switch (formerly BNT).

Problem:

1. HS22V blade running ESXi 5.1
2. Mount ESXi 5.5 Update 1 Driver Rollup 2 ISO image
3. Upgrade ESXi 5.1 to 5.5
4. Reboot the blade
5. All networking on the 10Gb vmnic appears OK (module loads, 10Gb links come up, CDP works, vSwitches report working uplinks), but traffic is not carried.  vmkping from any VMkernel interfaces bound to the Emulex adapter will fail, you can only ping yourself.
6. Run the command “esxcli software vib list | grep elxnet” and verify that the “elxnet” driver version is 10.0.783.13.
7. Run the command “esxcli network nic list” and verify that the 10Gb vmnics use the driver “elxnet” (new native driver).

Solution:

1. Download elxnet driver version 10.0.575.9.
2. Downgrade the elxnet driver from 10.0.783.13 to 10.0.575.9 using the command “esxcli software vib install –depot=<path>”.
3. Reboot ESXi.
4. The 10Gb interfaces will now carry traffic, test this with “vmkping <remote IP Address>”.

Background:

In ESXi 5.1 the “vmklinux” driver, “be2net” is used for Emulex adapters.  With the upgrade to ESXi 5.5, the new “native” driver, “elxnet” is used instead, also described in documentation available from Emulex.  The driver installed for Emulex adapter OCm10102-F-X (IBM P/N 49Y4234, Firmware version 4.0.1062.0) by the upgrade process is a supported driver, but does not function.  I opened a case with VMware GSS, who have now opened a PR for VMware Engineering to investigate.