在ESXi中启用Intel E810 25Gb网卡RoCEv2协议的方法

默认情况下，在ESXi 7.U3中Intel E810使用iWARP协议，如下图所示：

启用RoCEv2的步骤如下：

1.检查网卡驱动的RDMA参数，默认是启用的。如果未启用，使用“esxcli system module parameters set -m icen -p “RDMA=1,1””命令启用RDMA。

 esxcli system module parameters list -m icen 
Name         Type          Value  Description
-----------  ------------  -----  -----------
DRSS         array of int         Number of queues for Default Queue Receive-Side Scaling: 0/4/8/16 (default = 4)
FWLogEnable  array of int         Enable firmware logging functionality: 0 = Disable, 1 = Enable (default = 0)
FWLogEvents  long                 Firmware events to log. Binary math required to set. (Turn on Events 1 and 5 with input FWLogEvents = 17) (default = 0)
FWLogLevel   int                  FW event log level. Values: 0 = none, 1 = error, 2 = warning, 3 = normal, 4 = verbose. Invalid values: >= 5 (default = 0)
LLDP         array of int         Link Layer Discovery Protocol (LLDP) agent: 0 = disable, 1 = enable (default = 0)
MaxRdmaInts  int                  Maximum RDMA Interrupts (0..112) (default = 16)
NumQPsPerVF  array of int         Maximum number of queue pairs to be allocated for each VF in the PF (1/2/4/8/16) (default = 4)
RDMA         array of int         Enable Remote Direct Memory Access (RDMA) support, 0 = disable, 1 = enable (default = 1)
RSS          array of int         State of NetQueue Receive-Side Scaling: 0 = Disable, 1 = Enable (default = 1, supports RSS on multiple NetQueues)
RxITR        array of int         RX interrupt interval in microseconds, (-1 = Dynamic ITR, 0-8160 usec = Static ITR) (default = Dynamic ITR)
TxITR        array of int         TX interrupt interval in microseconds, (-1 = Dynamic ITR, 0-8160 usec = Static ITR) (default = Dynamic ITR)
VMDQ         array of int         Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default = 8)
max_vfs      array of int         Maximum number of VFs to be enabled (0..256)

2.使用命令启用RoCEv2协议。
1）检查irdman驱动的ROCE参数。
esxcli system module parameters list -m irdman

Name                    Type          Value  Description
----------------------  ------------  -----  -----------
DebugFlags              int                  debug flags: 0=disabled (default), 0x7fffffff=all
ROCE                    array of int         Enable RoCE support 0 = disable, 1 = enable, (default = 0)
ReducedMaxQpInitRdAtom  int                  Enable limit maxQpInitRdAtom to 0x10: 0=Disable (default), 1=Enable

2）启用RoCEv2协议。
esxcli system module parameters set -m irdman -p “ROCE=1,1”

3）确认参数设置成功。
esxcli system module parameters list -m irdman

Name                    Type          Value  Description
----------------------  ------------  -----  -----------
DebugFlags              int                  debug flags: 0=disabled (default), 0x7fffffff=all
ROCE                    array of int  1,1    Enable RoCE support 0 = disable, 1 = enable, (default = 0)
ReducedMaxQpInitRdAtom  int                  Enable limit maxQpInitRdAtom to 0x10: 0=Disable (default), 1=Enable

4）重启ESXi，让配置生效。
esxcli system shutdown reboot -r “reboot”

5）重新启动后，RDMA适配器的RoCEv2协议显示已启用。

来源：
ENABLING RDMA FOR VSAN WITH INTEL E810 ADAPTER