I’ve been running several network intensive services (Exchange 2010, SQL 2008R2, VMWare ESX) on Proliant DL385 G7 servers with HP’s 10Gb/s NC522SFP NICs for roughly 2 years and I’ve never experienced drivers and firmware with the same magnitude of bugs before.
We hit one recently we updated the drivers/firmware to fix a port flapping bug, which introduced a new one were network throughput slowed down from about 2.5Gb/s (on a 10Gb/s NIC) to roughly 400Kbps (occasionally hitting about 1Mb/s). Downgrading the drivers did not fix the issue (odd right?). Eventually found work around in HP forum. Apparently there’s also an updated driver for the NIC, but it’s not actually listed on the product support page, and this is a fairly obvious, and crippling bug.
From the HP forums: VERY slow network after man HW updates on DL370 G6
Changing the setting “Large Receive Offload” to “Disabled” as mentioned in the forum post, completely resolves the slowness, and doesn’t require a reboot.
But how could someone miss that when QAing a release? Someone saw a link light and thought to themselves, “Well that’s good enough, right?”
There is a long list of bad behaviors exhibited by these NICs which have been fixed by various firmware or driver updates. And this list is abnormally long in my opinion for any device.
There’s a full list at HP Support Site NC522SFP Troubleshooting documents, but below are the ones that have had the most impact for me personally.
- current – Driver update causes slow throughput when receiving data on NC522SFP NIC (as described above)
- 2012-09-07 HP NC522SFP Dual Port 10GbE Gigabit Server Adapter – NIC Goes Offline Breaking the NIC Team
- 2011-12-14 The HP QLogic P3 Online Firmware Upgrade Utility for Windows Server 2003/2008 May Fail With a “Dependency Failed” Error Message (my fave, couldn’t update firmware unless logged on as THE Administrator account, not just a user who had admin rights)
- 2011-08-11 NC522SFP … Network Adapters – FIRMWARE UPGRADE REQUIRED to Avoid the Loss and Automatic Recovery of Ethernet Connectivity or Adapter Unresponsiveness
It generally bugs me when people focus on buzzwords in meetings. Worse than that, the “Low Hanging Fruit” camp would like to do nothing more than Ironically spend the entire time figuring out what are all the easiest things to “achieve” and ignoring what’s the most important.
I heard someone mention in reference to the LHF mantra, “You know that fruit isn’t ripe, right?” Leading me to attempt to verify the statement and led me to a fantastic article about the innacuracy of the statement in the first place.