本文以ESXI6.7-CX3的组合做示例
mellanox官网下载对应的驱动安装包
ConnectX® Ethernet Driver for VMware® ESXi Server
下载后的文件
MLNX-NATIVE-ESX-ConnectX-3_3.17.70.1-1OEM.670.0.0-15491048.zip
参考手册: Installation - ESXi 6.7 v4.17.15.16 - NVIDIA Networking Docs
将文件解压成
MLNX-NATIVE-ESX-ConnectX-3_3.17.70.1-1OEM.670.0.0-offline_bundle-15491048.zip
将解压后的文件上传到ESXI上,使用SSH登录到ESXI里
vib install -d /<路径>/MLNX-NATIVE-ESX-ConnectX-3_3.17.70.1-1OEM.670.0.0-offline_bundle-15491048.zip
重启机器
[root@localhost:/vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB] esxcli software
vib install -d /vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB/MLNX-NATIVE-ESX-Co
nnectX-3_3.17.70.1-1OEM.670.0.0-offline_bundle-15491048.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: MEL_bootbank_nmlx4-core_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-en_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-rdma_3.17.70.1-1OEM.670.0.0.8169922
VIBs Removed: VMW_bootbank_nmlx4-core_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-en_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-rdma_3.17.9.12-1vmw.670.0.0.8169922
VIBs Skipped:
[root@localhost:/vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB] reboot
下载mellanox控制程序
Mellanox Firmware Tools (MFT) (nvidia.com)
安装对应版本
参考手册: Virtualization - ESXi 6.7 v4.17.15.16 - NVIDIA Networking Docs
vib install -v /<路径>/nmst-4.18.0.106-1OEM.650.0.0.4598673.x86_64.vib
vib install -v /<路径>/mft-4.18.0.106-10EM-600.0.0.4598673.x86_64.vib
[root@localhost:~] esxcli software
vib install -v /vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB/nmst-4.18.0.106-1O
EM.650.0.0.4598673.x86_64.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: MEL_bootbank_nmst_4.18.0.106-1OEM.650.0.0.4598673
VIBs Removed:
VIBs Skipped:
[root@localhost:~] esxcli software
vib install -v /vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB/mft-4.18.0.106-10E
M-600.0.0.4598673.x86_64.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: MEL_bootbank_mft_4.18.0.106-0
VIBs Removed:
VIBs Skipped:
[root@localhost:~] reboot
启动服务
[root@localhost:~] /opt/mellanox/bin/mst start
[root@localhost:~] /opt/mellanox/bin/mst status
MST devices:
------------
mt4099_pciconf0
mt4099_pci_cr0
查看当前拆分状态
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 q
Device #1:
----------
Device type: ConnectX3
Device: mt4099_pciconf0
Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 8
WOL_MAGIC_EN_P1 True(1)
LINK_TYPE_P1 VPI(3)
LINK_TYPE_P2 VPI(3)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)
观察这2个参数 SRIOV_EN = True(1) NUM_OF_VFS = 8 如果没有打开可以使用命令
[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
[root@localhost:~] reboot
使用max_vfs模块参数设置需要为 PF 创建的虚函数数。
[root@localhost:~] esxcli system module parameters set -m mlx4_core -p "max_vfs=8"