Missing Broadcom 5719 tg3 NICs after updates

Just updated one of my R720 ESXi 5.0 hosts. I did the firmware via the Dell OM_711_SUU_FULL_ISO_A00.iso which included things like BIOS 1.3.6 but also updated the Broadcom 5719 NIC firmware. The old firmware was version 7.2.14 and the new is 7.2.20.

I then updated the host with 14 updates via VMware Update Manager, one of which was the VMware ESXi 5.0 Complete Update 2 (ESXi500-Update02 – KB2033751), but another was Updates VMware ESXi 5.0 net-tg3 vib (ESXi500-201212210-UG – KB2033752).

Upon rebooting the host, all the 5719/tg3 vmnics were missing from the Configuration – Hardware – Network Adapters section in vSphere Client. Not good.

I thought I’d try unloading and reloading the tg3 driver so I ran the following commands via SSH on the host:

~ # vmkload_mod -u tg3
 vmkload_mod: Can not remove module tg3: Module not found
 ~ # vmkload_mod tg3
 Module tg3 loaded successfully

So the tg3 driver wasn’t loaded, but did load when I asked it to. Now I’m seeing all the vmnics in the vSphere Client. But will they stick after a reboot?

No. I SSH’d back into the host and ran

~ # esxcfg-module -l
 tg3                      Not Loaded

Had a look in /var/log/syslog.log and found eight of these messages each time the host boots:

2013-02-04T12:57:37Z jumpstart: VmkCtl: Loading module tg3 failed. Unable to load module /usr/lib/vmware/vmkmod/tg3: Bad parameter

Per KB1038247, I had a look for “tg3” in /etc/vmware/esx.conf and found the line:

/vmkernel/module/tg3/options = "force_netq=0,0,0,0,0,0,0,0"

which relates to the current issues with the tg3 driver and NetQueue. Alternatively you can use:

~ # esxcfg-module -g tg3
 tg3 enabled = 1 options = 'force_netq=0,0,0,0,0,0,0,0'

The KB article says that editing the esx.conf file by hand is a bad idea so use the command:

~ # esxcfg-module -s "" tg3
 ~ # esxcfg-module -g tg3
 tg3 enabled = 1 options = ''

and reboot. Will it load now? Yes, I’m seeing all the vmnics I’d expect.

So does the newer 3.123b.v50.1 tg3 driver not like the force_netq option? I just issued the following command to get the driver version on a host that I’ve not applied the ESXi500-201212210-UG patch to, and it’s showing the same driver version!

~ # ethtool -i vmnic4
driver: tg3
version: 3.123b.v50.1
firmware-version: FFV7.2.14 bc 5719-v1.29
bus-info: 0000:43:00.0

Yet when I scan the host with VUM it is showing it as “Missing” the ESXi500-201212210-UG patch. Erm? Perhaps VUM looks at the patch rather than the driver version within the patch. But if the driver version before and after the patch install is the same, why is the new driver not starting with the force_netq option present?

The correct way to test this would be to put the option back and reboot. I’m not sure I can be bothered today, this wasn’t supposed to take all day… I think I might try the alternative suggestion and just disable NetQueue for the entire host by using:

esxcfg-advcfg -k FALSE netNetqueueEnabled

as I just noticed that (according to KB2035701) NetQueue is pointless with 1Gb NICs anyway, and whilst I do have two 10Gb NICs in these hosts, they’re not currently connected to anything.

Update 2013-02-05: I’ve just tried installing the tg3 “update” patch onto another host via VUM without doing any other updates, and all the tg3 vmnics have dissappeared. So that seems to be the culprit.

I should point out that I originally installed ESXi onto these R720 servers by using the Dell ESXi installer VMware-VMvisor-Installer-5.0.0.update1-623860.x86_64-Dell_Customized_RecoveryCD_A00.iso which must be where I got the 3.123b.v50.1 tg3 driver from.

This entry was posted in vSphere and tagged , , , , , , , , , , , , , , . Bookmark the permalink.

5 Responses to Missing Broadcom 5719 tg3 NICs after updates

  1. Christian says:

    Thanks for sharing! :-)

    I was really happy to find your post, as I ran into this very issue tonight, while applying vSphere ESXi 5.0 Update 2 on our hosts. Saved me a lot of trial and error, and took me straight to the point.

    Checking the driver details on my servers before and after, revealed that the tg3 driver is indeed updated. Even if the version number is the same, the build number and date is different.

    You can check the installed driver with this command:
    vmkload_mod -s tg3|grep Version

    In my case the existing driver (for which the “force_netq=0,0,0,0” option worked) showed:
    Version: Version 3.123b.v50.1, Build: 472560, Interface: 9.2 Built on: Apr 3 2012

    Whereas the driver provided with the Update 2 updates showed:
    Version: Version 3.123b.v50.1, Build: 914586, Interface: 9.2 Built on: Nov 18 2012

  2. rcmtech says:

    Comment from Broadcom: VMware has apparently removed NetQueue support from the inbox tg3 driver (probably for the same rationale as documented in KB2035701), so they have removed the force_netq module parameter as well. This is the reason the tg3 driver load fails with ‘Bad parameter’ when the force_netq option is present in the esx.conf file.

  3. Pingback: VMware ESX/ESXi issues with Broadcom 5719 quad port Gb NIC | Robin CM's IT Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s