Archive | March 2013

WHO QA’d THAT?: HP NC522SFP driver/firmware bugs

I’ve been running several network intensive services (Exchange 2010, SQL 2008R2, VMWare ESX) on Proliant DL385 G7 servers with HP’s 10Gb/s NC522SFP NICs for roughly 2 years and I’ve never experienced drivers and firmware with the same magnitude of bugs before.

We hit one recently we updated the drivers/firmware to fix a port flapping bug, which introduced a new one were network throughput slowed down from about 2.5Gb/s (on a 10Gb/s NIC) to roughly 400Kbps (occasionally hitting about 1Mb/s).  Downgrading the drivers did not fix the issue (odd right?).  Eventually found work around in HP forum.  Apparently there’s also an updated driver for the NIC, but it’s not actually listed on the product support page, and this is a fairly obvious, and crippling bug.

From the HP forums: VERY slow network after man HW updates on DL370 G6

Changing the setting “Large Receive Offload” to “Disabled” as mentioned in the forum post, completely resolves the slowness, and doesn’t require a reboot.

But how could someone miss that when QAing a release?  Someone saw a link light and thought to themselves, “Well that’s good enough, right?”

There is a long list of bad behaviors exhibited by these NICs which have been fixed by various firmware or driver updates.  And this list is abnormally long in my opinion for any device.

There’s a full list at HP Support Site NC522SFP Troubleshooting documents, but below are the ones that have had the most impact for me personally.

Low Hanging Fruit, a term I’ve come to loathe.

It generally bugs me when people focus on buzzwords in meetings. Worse than that, the “Low Hanging Fruit” camp would like to do nothing more than Ironically spend the entire time figuring out what are all the easiest things to “achieve” and ignoring what’s the most important.

I heard someone mention in reference to the LHF mantra, “You know that fruit isn’t ripe, right?” Leading me to attempt to verify the statement and led me to a fantastic article about the innacuracy of the statement in the first place.