I just came across this issue at a customer site, so I will share the solution for those who might get into this bug.
Symptoms : Cannot change license on the HP ESXi 5.0 Aug 2011 ESXi HD-USB-SDImgeInstlr (Z7550-00204.iso), and once you try, the connection to vCenter is highly unstable, going into a “Not Responding” behavior.
Solution : HP has currently documented the process to fix this behavior.
1) Download the VIB file here : http://vibsdepot.hp.com/hp-esxi-5.0-license/hp-esx-license-1.0-5.zip
wget is available in ESXi 5 (small download utility), so if your host is connected to internet, you can get it quite easily.
2) Login into the local console on the affected host.
3) Copy the VIB to a temporary location on the affected host.
4) Apply the new license VIB by executing the command
5) Reboot the host
HP will soon rebuild a new build with this fix included.
You can also encounter this issue, on the be2net network adapter driver : http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007397.