WHO QA’d THAT?: HP NC522SFP driver/firmware bugs
I’ve been running several network intensive services (Exchange 2010, SQL 2008R2, VMWare ESX) on Proliant DL385 G7 servers with HP’s 10Gb/s NC522SFP NICs for roughly 2 years and I’ve never experienced drivers and firmware with the same magnitude of bugs before.
We hit one recently we updated the drivers/firmware to fix a port flapping bug, which introduced a new one were network throughput slowed down from about 2.5Gb/s (on a 10Gb/s NIC) to roughly 400Kbps (occasionally hitting about 1Mb/s). Downgrading the drivers did not fix the issue (odd right?). Eventually found work around in HP forum. Apparently there’s also an updated driver for the NIC, but it’s not actually listed on the product support page, and this is a fairly obvious, and crippling bug.
From the HP forums: VERY slow network after man HW updates on DL370 G6
Changing the setting “Large Receive Offload” to “Disabled” as mentioned in the forum post, completely resolves the slowness, and doesn’t require a reboot.
But how could someone miss that when QAing a release? Someone saw a link light and thought to themselves, “Well that’s good enough, right?”
There is a long list of bad behaviors exhibited by these NICs which have been fixed by various firmware or driver updates. And this list is abnormally long in my opinion for any device.
There’s a full list at HP Support Site NC522SFP Troubleshooting documents, but below are the ones that have had the most impact for me personally.
- current – Driver update causes slow throughput when receiving data on NC522SFP NIC (as described above)
- 2012-09-07 HP NC522SFP Dual Port 10GbE Gigabit Server Adapter – NIC Goes Offline Breaking the NIC Team
- 2011-12-14 The HP QLogic P3 Online Firmware Upgrade Utility for Windows Server 2003/2008 May Fail With a “Dependency Failed” Error Message (my fave, couldn’t update firmware unless logged on as THE Administrator account, not just a user who had admin rights)
- 2011-08-11 NC522SFP … Network Adapters – FIRMWARE UPGRADE REQUIRED to Avoid the Loss and Automatic Recovery of Ethernet Connectivity or Adapter Unresponsiveness