iSCSI – Hardware or Software – How many TOEs do you have?

More and more of my new implementations of VMware Infrastructure are being connected to iSCSI SANs (EMC, LeftHand, and Equallogic) and the question has come up about whether or not to spend extra dollars on TOE (TCPIP Offload Engine) Network cards.  The TOE cards take the burden of processing iSCSI packets away from the host’s CPU and place it on the Network Interface Card itself.  In theory this should speed performance of the host machine.  For ESX implementations, VMware has provided a very solid software iSCSI initiator that I have been using with great regularity.  I’ve become curious if others are using TOE cards in their environments or just using the straight SW solution.

In the past (ESX 3.0.2), VMware only officially supported 2 TOE Cards in their hardware I/O compatibility guide which lead me to believe that TOE cards were not widely adopted yet.  A quick scan of VMware’s new Searchable Hardware Compatibility Guide reveals that about 16 TOE cards are now supported so acceptance rates might have changed.

Most of the compatible cards seem to be in the $1000 dollar range while most 1GB Network cards seem to be in the $200 range.  Due to the high costs of these TOE Cards, I am curious to know if anyone is receiving the expected performance increases to justify the higher costs and if it is at all worth the price and effort of putting the TOE cards in.

If you have any experiences with TOE cards in ESX environments, please let me know your thoughts in the comments.  Thanks!