The kind folks over at Nimble Storage were nice enough to drop me a note this evening regarding an issue regarding VMware e1000e vNIC and iSCSI attached storage on Microsoft Windows Server 2012. The field alert states that:
VMware has confirmed a severe issue that may cause data inconsistency on iSCSI storage systems under the following conditions:
NOTE: This issue is not specific to Nimble Storage
1. Guest OS: Windows 2012 (Windows 8 may also be suspect)
2. Guest NIC emulation type: e1000e
3. TCP segmentation offload (TSO) is enabled
4. ESX 5.0/5.1
VMware says this about the E1000e vNIC as:
E1000e: This feature emulates a newer model of Intel Gigabit NIC (number 82574) in the virtual hardware. This is known as the “e1000e” vNIC. e1000e is available only on hardware version 8 (and newer) virtual machines in vSphere 5. It is the default vNIC for Windows 8 and newer (Windows) guest operating systems.
The fact that this is most likely the default vNIC configured on many newer VM‘s makes this issue all that more important and worth your review. Nimble Storage listed the following preventative actions and next steps:
1. We recommend you immediately stop using e1000e virtual emulation in all cases and use one of the other virtual NICs provided in the guest Virtual Machine.
e.g. VMXNET 3 or e1000 (known to NOT have this issue)
1. There is no change necessary to Nimble OS. This issue is not specific to Nimble Storage; this issue has been reproduced on other manufacturers iSCSI Storage Arrays.
2. If you are using the configuration shown above, please take necessary steps to confirm the integrity of all data on the iSCSI volumes attached in this manner.
It is VERY important to note that this has been confirmed against multiple storage arrays so this is, as stated, NOT simply a Nimble Storage issue. This is a VMware issue and they have addressed it in the VMware KB2058692 where they state the symptoms as:
On a Windows 2012 virtual machine using the default e1000e network adapter and running on an ESXi 5.0/5.1 host, you experience these symptoms:
Data corruption may occur when copying data over the network.
Data corruption may occur after a network file copy event.