All hardware is on the HCL. The 1GB NIC is a few driver versions back but the 10GB NIC has the latest driver installed and vMotion over that produces the same random issues. Looking at one of the failed VM in the vmkernel.log I don't see any errors:
Source Host:
2013-06-05T15:04:31.431Z cpu5:5282792)Migrate: vm 5282793: 3234: Setting VMOTION info: Source ts = 1370444633537231, src ip = <10.100.12.206> dest ip = <10.100.12.205> Dest wid = 400487 using SHARED swap
2013-06-05T15:04:31.433Z cpu5:5282792)Tcpip_Vmk: 1059: Affinitizing 10.100.12.206 to world 6541311, Success
2013-06-05T15:04:31.433Z cpu5:5282792)VMotion: 2425: 1370444633537231 S: Set ip address '10.100.12.206' worldlet affinity to send World ID 6541311
Destination Host:
2013-06-05T15:04:31.196Z cpu8:400447)World: vm 400487: 1421: Starting world vmm0:LNMTCODCPD20 with flags 8
2013-06-05T15:04:31.196Z cpu8:400447)Sched: vm 400487: 6416: Adding world 'vmm0:LNMTCODCPD20', group 'host/user/pool2', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=2097152 minLimit=-1 max=-1
2013-06-05T15:04:31.196Z cpu8:400447)Sched: vm 400487: 6431: renamed group 695847 to vm.400447
2013-06-05T15:04:31.196Z cpu8:400447)Sched: vm 400487: 6448: group 695847 is located under group 695838
2013-06-05T15:04:31.311Z cpu8:400447)Migrate: vm 400487: 3273: Setting VMOTION info: Dest ts = 1370444633537231, src ip = <10.100.12.206> dest ip = <10.100.12.205> Dest wid = 0 using SHARED swap
2013-06-05T15:04:31.315Z cpu8:400447)Hbr: 3308: Migration start received (worldID=400487) (migrateType=1) (event=0) (isSource=0) (sharedConfig=1)
2013-06-05T15:05:40.766Z cpu14:400447)VSCSI: 3780: handle 8222(vscsi0:0):Creating Virtual Device for world 400487 (FSS handle 3226637)
2013-06-05T15:05:41.034Z cpu7:400487)VMMVMKCall: 208: Received INIT from world 400487
2013-06-05T15:05:41.035Z cpu7:400487)LSI: 1986: LSI: Initialized rings for scsi0 async=1, record=0 replay=0
2013-06-05T15:05:41.039Z cpu7:400487)VMotion: 5679: 1370444633537231 D: Received all changed pages.
2013-06-05T15:05:41.044Z cpu7:400487)VmMemMigrate: vm 400487: 5005: Regular swap file bitmap checks out.
2013-06-05T15:05:41.056Z cpu7:400487)VMotion: 5458: 1370444633537231 D: Resume handshake successful
2013-06-05T15:05:41.240Z cpu8:400745)Hbr: 3405: Migration end received (worldID=400487) (migrateType=1) (event=1) (isSource=0) (sharedConfig=1)
2013-06-05T15:05:41.270Z cpu9:400751)Swap: vm 400487: 3254: Starting prefault for the migration swap file
2013-06-05T15:05:41.341Z cpu9:400751)Swap: vm 400487: 3429: Finish swapping in migration swap file. (faulted 0 pages, pshared 0 pages). Success.
2013-06-05T16:27:57.958Z cpu8:400487)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401d0d1c0, 0) to dev "mpx.vmhba35:C0:T0:L0" on path "vmhba35:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-06-05T16:27:57.958Z cpu8:400487)ScsiDeviceIO: 2331: Cmd(0x412401d0d1c0) 0x1a, CmdSN 0xb0b502 from world 0 to dev "mpx.vmhba35:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.