CentOS 7下CX5-RDMA网络测试

RDMA(Remote Direct Memory Access) 全称远程直接数据存取，就是为了解决网络传输中服务器端数据处理的延迟而产生的。RDMA 通过网络把资料直接传入计算机的存储区，将数据从一个系统快速移动到远程系统存储器中，而不对操作系统造成任何影响，这样就不需要用到多少计算机的处理功能。它消除了外部存储器复制和上下文切换的开销，因而能解放内存带宽和 CPU 周期用于改进应用系统性能。RDMA需要智能网卡支持，这里使用的是Mellanox cx5。

基于CentOS 7.8 x86_64

1. 识别CX5 网卡

#lspci |grep Mellanox
5e:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
5e:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

2. 安装MLNX驱动

选择下载与OS匹配的驱动包，地址：https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/。

官方文档：https://enterprise-support.nvidia.com/s/article/howto-install-mlnx-ofed-driver

#下载 MLNX_OFED_LINUX-23.10-3.2.2.0-rhel7.8-x86_64.tgz
#yum install createrepo
#yum install tcl fuse-libs tk
#tar zxvf MLNX_OFED_LINUX-23.10-3.2.2.0-rhel7.8-x86_64.tgz
#cd MLNX_OFED_LINUX-23.10-3.2.2.0-rhel7.8-x86_64
#./mlnxofedinstall  --add-kernel-support --with-nvmf --force    #我这里后面要测试nvmeof rdma
#dracut -f
# /etc/init.d/openibd restart
Unloading HCA driver:                                      [  OK  ]
Loading HCA driver and Access Layer:                       [  OK  ]
#systemctl enable openibd
#reboot
# lsmod |grep -i nvme
nvme                   47306  8 
nvme_core              94686  5 nvme
mlx_compat             55285  13 nvme,rdma_cm,ib_cm,iw_cm,auxiliary,mlx5_ib,nvme_core,ib_core,ib_umad,ib_uverbs,mlx5_core,rdma_ucm,ib_ipoib

3. 检查设备

#ibdev2netdev
mlx5_0 port 1 ==> eth2 (Up)
mlx5_1 port 1 ==> eth3 (Up)
#ibstatus
Infiniband device 'mlx5_0' port 1 status:
        default gid:     fe80:0000:0000:0000:1270:fdff:fe27:f982
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            25 Gb/sec (1X EDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_1' port 1 status:
        default gid:     fe80:0000:0000:0000:1270:fdff:fe27:f983
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            25 Gb/sec (1X EDR)
        link_layer:      Ethernet

常用命令

• ibstat: 查询 InfiniBand 设备的基本状态
• ibstatus： 网卡信息
• ibv_devinfo：网卡设备信息（ibv_devinfo -d mlx5_0 -v）
• ibv_devices：查看本主机的 infiniband 设备
• ibnodes：查看网络中的 infiniband 设备
• show_gids：看看网卡支持的 roce 版本
• show_counters: 网卡端口统计数据，比如发送接受数据大小
• mlxconfig: 网卡配置（mlxconfig -d mlx5_1 q 查询网卡配置信息）

吞吐量测试

写吞吐量

在 RDMA 驱动安装时会安装一些 RDMA 工具，可以使用 ib_send_bw 测试写吞吐量

服务器 A（server）：

ib_write_bw -a -d mlx5_0

服务器 B（client）：

ib_write_bw -a -d mlx5_0 10.192.51.152 (server端ip)

读吞吐量

读吞吐量的测试与写吞吐量测试相同，只是使用命令换为 ib_read_bw

延时测试

测试同样分为读写，测试工具为 ib_read_lat、ib_write_lat

• Performance Tuning for Mellanox Adapters

延时测试

测试同样分为读写，测试工具为 ib_read_lat、ib_write_lat

• Performance Tuning for Mellanox Adapters

带宽统计

在使用 RDMA 时，发送和接收的数据带宽可以在 app 中自己进行收集，这样我们的程序发送和接收的数据量会很清楚。
如果想知道当前 RDMA 网卡所发送和接收的带宽可以通过 sysfs 下的相关节点获取。

• 发送数据量（byte）：/sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data
• 接收数据量（byte）：/sys/class/infiniband/mlx5_0/ports/1/counters/port_rcv_data

注：port_xmit_data 和 port_rcv_data 的数值是实际的 1/4, 因此实际的带宽是在其基础之上乘以 4，应该是为了防止数据溢出

port_xmit_data: (RO) Total number of data octets, divided by 4 (lanes), transmitted on all VLs. This is 64 bit counter
port_rcv_data: (RO) Total number of data octets, divided by 4 (lanes), received on all VLs. This is 64 bit counter.

来自： Documentation/ABI/stable/sysfs-class-infiniband

pma_cnt_ext->port_xmit_data =
    cpu_to_be64(MLX5_SUM_CNT(out, transmitted_ib_unicast.octets,
                 transmitted_ib_multicast.octets) >> 2);
pma_cnt_ext->port_rcv_data =
    cpu_to_be64(MLX5_SUM_CNT(out, received_ib_unicast.octets,
                 received_ib_multicast.octets) >> 2);

file: drivers/infiniband/hw/mlx5/mad.c

网络联通性测试

由于当前网卡只支持 Ethernet 模式，因此只能使用 ibv_rc_pingpong 进行 ping 测试。

• https://community.mellanox.com/s/article/RoCE-Debug-Flow-for-Linux

服务器A

# ibv_rc_pingpong -d mlx5_0 -g 0
  local address:  LID 0x0000, QPN 0x000088, PSN 0xf4799b, GID fe80::1270:fdff:fe27:f982
  remote address: LID 0x0000, QPN 0x000088, PSN 0x22fd0b, GID fe80::ac0:ebff:fef4:4bf4
8192000 bytes in 0.01 seconds = 6475.25 Mbit/sec
1000 iters in 0.01 seconds = 10.12 usec/iter

client B

# ibv_rc_pingpong -d mlx5_1 -g 0 10.192.51.152
  local address:  LID 0x0000, QPN 0x000088, PSN 0x22fd0b, GID fe80::ac0:ebff:fef4:4bf4
  remote address: LID 0x0000, QPN 0x000088, PSN 0xf4799b, GID fe80::1270:fdff:fe27:f982
8192000 bytes in 0.01 seconds = 6746.55 Mbit/sec
1000 iters in 0.01 seconds = 9.71 usec/iter

counters

# ls -lsh /sys/class/infiniband/mlx5_0/ports/1/counters/
total 0
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 excessive_buffer_overrun_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 link_downed
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 link_error_recovery
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 local_link_integrity_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 multicast_rcv_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 multicast_xmit_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_rcv_constraint_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_rcv_data
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_rcv_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_rcv_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_rcv_remote_physical_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_rcv_switch_relay_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_xmit_constraint_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_xmit_data
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_xmit_discards
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_xmit_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 port_xmit_wait
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 symbol_error
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 unicast_rcv_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 unicast_xmit_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 VL15_dropped

Counter Description:

Counter	Description	InfiniBand Spec Name	Group
port_rcv_data	The total number of data octets, divided by 4, (counting in double words, 32 bits), received on all VLs from the port.	PortRcvData	Informative
port_rcv_packets	Total number of packets (this may include packets containing Errors. This is 64 bit counter.	PortRcvPkts	Informative
port_multicast_rcv_packets	Total number of multicast packets, including multicast packets containing errors.	PortMultiCastRcvPkts	Informative
port_unicast_rcv_packets	Total number of unicast packets, including unicast packets containing errors.	PortUnicastRcvPkts	Informative
port_xmit_data	The total number of data octets, divided by 4, (counting in double words, 32 bits), transmitted on all VLs from the port.	PortXmitData	Informative
port_xmit_packetsport_xmit_packets_64	Total number of packets transmitted on all VLs from this port. This may include packets with errors.This is 64 bit counter.	PortXmitPkts	Informative
port_rcv_switch_relay_errors	Total number of packets received on the port that were discarded because they could not be forwarded by the switch relay.	PortRcvSwitchRelayErrors	Error
port_rcv_errors	Total number of packets containing an error that were received on the port.	PortRcvErrors	Informative
port_rcv_constraint_errors	Total number of packets received on the switch physical port that are discarded.	PortRcvConstraintErrors	Error
local_link_integrity_errors	The number of times that the count of local physical errors exceeded the threshold specified by LocalPhyErrors.	LocalLinkIntegrityErrors	Error
port_xmit_wait	The number of ticks during which the port had data to transmit but no data was sent during the entire tick (either because of insufficient credits or because of lack of arbitration).	PortXmitWait	Informative
port_multicast_xmit_packets	Total number of multicast packets transmitted on all VLs from the port. This may include multicast packets with errors.	PortMultiCastXmitPkts	Informative
port_unicast_xmit_packets	Total number of unicast packets transmitted on all VLs from the port. This may include unicast packets with errors.	PortUnicastXmitPkts	Informative
port_xmit_discards	Total number of outbound packets discarded by the port because the port is down or congested.	PortXmitDiscards	Error
port_xmit_constraint_errors	Total number of packets not transmitted from the switch physical port.	PortXmitConstraintErrors	Error
port_rcv_remote_physical_errors	Total number of packets marked with the EBP delimiter received on the port.	PortRcvRemotePhysicalErrors	Error
symbol_error	Total number of minor link errors detected on one or more physical lanes.	SymbolErrorCounter	Error
VL15_dropped	Number of incoming VL15 packets dropped due to resource limitations (e.g., lack of buffers) of the port.	VL15Dropped	Error
link_error_recovery	Total number of times the Port Training state machine has successfully completed the link error recovery process.	LinkErrorRecoveryCounter	Error
link_downed	Total number of times the Port Training state machine has failed the link error recovery process and downed the link.	LinkDownedCounter	Error

hw_counters

# ls -lsh /sys/class/infiniband/mlx5_0/ports/1/hw_counters/
total 0
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 duplicate_request
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 implied_nak_seq_err
0 -rw-r--r-- 1 root root 4.0K 5月  28 16:42 lifespan
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 local_ack_timeout_err
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 np_cnp_sent
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 np_ecn_marked_roce_packets
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 out_of_buffer
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 out_of_sequence
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 packet_seq_err
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 req_cqe_error
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 req_cqe_flush_error
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 req_remote_access_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 req_remote_invalid_request
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 resp_cqe_error
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 resp_cqe_flush_error
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 resp_local_length_error
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 resp_remote_access_errors
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rnr_nak_retry_err
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rp_cnp_handled
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rp_cnp_ignored
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rx_atomic_requests
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rx_icrc_encapsulated
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rx_read_requests
0 -r--r--r-- 1 root root 4.0K 5月  24 15:28 rx_write_requests

HW Counters Description:

Counter	Description	Group
duplicate_request	Number of received packets. A duplicate request is a request that had been previously executed.	Error
implied_nak_seq_err	Number of time the requested decided an ACK. with a PSN larger than the expected PSN for an RDMA read or response.	Error
lifespan	The maximum period in ms which defines the aging of the counter reads. Two consecutive reads within this period might return the same values	Informative
local_ack_timeout_err	The number of times QP’s ack timer expired for RC, XRC, DCT QPs at the sender side.The QP retry limit was not exceed, therefore it is still recoverable error.	Error
np_cnp_sent	The number of CNP packets sent by the Notification Point when it noticed congestion experienced in the RoCEv2 IP header (ECN bits).The counters was added in MLNX_OFED 4.1	Informative
np_ecn_marked_roce_packets	The number of RoCEv2 packets received by the notification point which were marked for experiencing the congestion (ECN bits where ‘11’ on the ingress RoCE traffic) .The counters was added in MLNX_OFED 4.1	Informative
out_of_buffer	The number of drops occurred due to lack of WQE for the associated QPs.	Error
out_of_sequence	The number of out of sequence packets received.	Error
packet_seq_err	The number of received NAK sequence error packets. The QP retry limit was not exceeded.	Error
req_cqe_error	The number of times requester detected CQEs completed with errors.The counters was added in MLNX_OFED 4.1	Error
req_cqe_flush_error	The number of times requester detected CQEs completed with flushed errors.The counters was added in MLNX_OFED 4.1	Error
req_remote_access_errors	The number of times requester detected remote access errors.The counters was added in MLNX_OFED 4.1	Error
req_remote_invalid_request	The number of times requester detected remote invalid request errors.The counters was added in MLNX_OFED 4.1	Error
resp_cqe_error	The number of times responder detected CQEs completed with errors.The counters was added in MLNX_OFED 4.1	Error
resp_cqe_flush_error	The number of times responder detected CQEs completed with flushed errors.The counters was added in MLNX_OFED 4.1	Error
resp_local_length_error	The number of times responder detected local length errors.The counters was added in MLNX_OFED 4.1	Error
resp_remote_access_errors	The number of times responder detected remote access errors.The counters was added in MLNX_OFED 4.1	Error
rnr_nak_retry_err	The number of received RNR NAK packets. The QP retry limit was not exceeded.	Error
rp_cnp_handled	The number of CNP packets handled by the Reaction Point HCA to throttle the transmission rate.The counters was added in MLNX_OFED 4.1	Informative
rp_cnp_ignored	The number of CNP packets received and ignored by the Reaction Point HCA. This counter should not raise if RoCE Congestion Control was enabled in the network. If this counter raise, verify that ECN was enabled on the adapter. See HowTo Configure DCQCN (RoCE CC) values for ConnectX-4 (Linux).The counters was added in MLNX_OFED 4.1	Error
rx_atomic_requests	The number of received ATOMIC request for the associated QPs.	Informative
rx_dct_connect	The number of received connection request for the associated DCTs.	Informative
rx_read_requests	The number of received READ requests for the associated QPs.	Informative
rx_write_requests	The number of received WRITE requests for the associated QPs.	Informative
rx_icrc_encapsulated	The number of RoCE packets with ICRC errors.This counter was added in MLNX_OFED 4.4 and kernel 4.19	Error
roce_adp_retrans	Counts the number of adaptive retransmissions for RoCE trafficThe counter was added in MLNX_OFED rev 5.0-1.0.0.0 and kernel v5.6.0	Informative
roce_adp_retrans_to	Counts the number of times RoCE traffic reached timeout due to adaptive retransmissionThe counter was added in MLNX_OFED rev 5.0-1.0.0.0 and kernel v5.6.0	Informative
roce_slow_restart	Counts the number of times RoCE slow restart was usedThe counter was added in MLNX_OFED rev 5.0-1.0.0.0 and kernel v5.6.0	Informative
roce_slow_restart_cnps	Counts the number of times RoCE slow restart generated CNP packetsThe counter was added in MLNX_OFED rev 5.0-1.0.0.0 and kernel v5.6.0	Informative
roce_slow_restart_trans	Counts the number of times RoCE slow restart changed state to slow restartThe counter was added in MLNX_OFED rev 5.0-1.0.0.0 and kernel v5.6.0	Informative

• duplicate_request:（Duplicated packets）接收报文数，重复请求是先前已执行的请求。
• out_of_sequence:（Drop out of sequence）接收到的乱序包的数量，说明此时已经产生了丢包
• packet_seq_err：（NAK sequence rcvd）接收到的 NAK 序列错误数据包的数量，未超过 QP 重试限制。