ESXi直通Intel网卡接口出现reset故障解决

最近买了个intel 82576的双口网卡，想直通给虚拟机做软路由使用，设置好直通，虚拟机添加PCI设备后，开机后接口一直报reset错误，无法使用。

ESXI的直通设置如下


启动后报错如下

[   21.053132] igb 0000:0b:00.0 eth1: Reset adapter
[   22.076862] igb 0000:0b:00.1 eth2: Reset adapter
[   23.724403] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   25.149404] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   29.244841] igb 0000:0b:00.0 eth1: Reset adapter
[   31.292822] igb 0000:0b:00.1 eth2: Reset adapter
[   31.858408] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   34.543434] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   36.924903] igb 0000:0b:00.0 eth1: Reset adapter
[   39.525468] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   39.996899] igb 0000:0b:00.1 eth2: Reset adapter
[   43.136467] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   45.116915] igb 0000:0b:00.0 eth1: Reset adapter
[   47.489434] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   49.212899] igb 0000:0b:00.1 eth2: Reset adapter
[   52.331477] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   52.796902] igb 0000:0b:00.0 eth1: Reset adapter
[   55.475477] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   57.916936] igb 0000:0b:00.1 eth2: Reset adapter
[   60.988918] igb 0000:0b:00.0 eth1: Reset adapter
[   61.037449] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   63.266458] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   67.132947] igb 0000:0b:00.1 eth2: Reset adapter
[   69.180902] igb 0000:0b:00.0 eth1: Reset adapter
[   70.309504] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   71.497487] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   75.836890] igb 0000:0b:00.1 eth2: Reset adapter
[   76.860913] igb 0000:0b:00.0 eth1: Reset adapter
[   78.921511] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   79.133500] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   85.052978] igb 0000:0b:00.0 eth1: Reset adapter
[   87.786511] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   89.148945] igb 0000:0b:00.1 eth2: Reset adapter
[   92.277536] igb 0000:0b:00.1 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   93.245009] igb 0000:0b:00.0 eth1: Reset adapter
[   95.605522] igb 0000:0b:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   97.854287] igb 0000:0b:00.1 eth2: Reset adapter
vyos@vyos:~$      

查看接口的相关信息如下

vyos@vyos:~$ sudo ethtool -i eth1
driver: igb
version: 5.10.103-amd64-vyos
firmware-version: 1.2.1
expansion-rom-version: 
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

vyos@vyos:~$ lspci -vvv
0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
        Physical Slot: 192
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at fd3c0000 (32-bit, non-prefetchable) [size=128K]
        Region 1: Memory at fcc00000 (32-bit, non-prefetchable) [size=4M]
        Region 2: I/O ports at 5000 [size=32]
        Region 3: Memory at fd3fc000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: igb
        Kernel modules: igb

0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
        Physical Slot: 192
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 16
        Region 0: Memory at fd3a0000 (32-bit, non-prefetchable) [size=128K]
        Region 1: Memory at fc800000 (32-bit, non-prefetchable) [size=4M]
        Region 2: I/O ports at 5020 [size=32]
        Region 3: Memory at fd3f8000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: igb
        Kernel modules: igb

已经是最新的intel驱动，查看半天也没有找到是什么问题。

查看了相关的文档并进行了测试，看到其他人也遇到类似的问题，在kvm的环境下直通网卡没有任何问题，在ESXI下就有问题。问题定位在VMware上。
看到VMware的官方文档说有可能跟MSI或MSI-X有关，具体也没仔细研究。
https://www.vmware.com/pdf/vsp_4_vmdirectpath_host.pdf
文档上说为虚机配置文件添加

pciPassthru0.msiEnabled = “FALSE”

但是我试了下，没有任何效果。还在一直报reset adapter错误。

继续查看相关文档，说在虚机启动的时候添加

pci=nomsi pci=pcie_scan_all

能解决问题。

测试后问题确实解决了。。。

本文发布只是为了记录解决生活中的遇到的问题，且为本人原创，转载请说明。

参考文章：
https://www.vmware.com/pdf/vsp_4_vmdirectpath_host.pdf
http://forum.tinycorelinux.net/index.php/topic,25002.0.html