【自记录】龙蜥8.2 RDMA（RXE软实现）配置

无边风月-风之羽翼

于 2024-04-03 17:59:18 发布

阅读量478

点赞数 3

分类专栏： OS 文章标签： rdma

本文链接：https://blog.csdn.net/gogoytgo/article/details/137352742

版权

OS 专栏收录该内容

29 篇文章 0 订阅

订阅专栏

1 前言

因为有需要了解RDMA，因此找了2台机器进行部署验证，理解细节。
RDMA默认是需要特殊网卡，甚至特殊交换机的，但是也可以基于RXE软件实现。因此本次就用RXE看看效果。

2 工具安装

希望使用软件模拟RDMA环境验证

yum install libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils perl-Switch elfutils-libelf-devel  -y

运行rxe_cfg可看到当前的网卡情况

$ rxe_cfg 
  Name             Link  Driver  Speed  NMTU  IPv4_addr      RDEV  RMTU                        
  eth0             no    igb            1500  192.168.1.100

启动RXE并将对外网卡加入

rxe_cfg start
rxe_cfg add eth0

此时重新执行rxe_cfg和ibv_devinfo 检查结果

$ rxe_cfg 
  Name             Link  Driver  Speed  NMTU  IPv4_addr      RDEV  RMTU                        
  eth0             yes   igb            1500  192.168.1.100  rxe0  1024  (3)        


$ ibv_devinfo 
hca_id: rxe0
        transport:                      InfiniBand (0)
        fw_ver:                         0.0.0
        node_guid:                      e695:6eff:fe2f:3591
        sys_image_guid:                 0000:0000:0000:0000
        vendor_id:                      0x0000
        vendor_part_id:                 0
        hw_ver:                         0x0
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

3 连通性验证

可使用rping进行验证

机器A（IP地址为192.168.1.100）扮演服务端，填写自身对外IP，运行

rping -s -a 192.168.1.100 -v -C 10

此时命令行进行等待状态。

接着在机器B（IP地址为192.168.1.200）上运行

rping -c -a 192.168.1.100 -v -C 10

可看到扮演服端的机器A上输出以下内容；客户端也有类似的内容，即表示联通

server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10

4 测速

测速需要安装额外的组件

yum install perftest -y

4.1 写速度测速

在扮演服务端的机器A上运行

ib_write_bw

进入等待状态。

在扮演客户端的机器B上填写机器A的IP，运行

ib_write_bw 192.168.1.100

可在任意一侧观看结果。两台机器间使用1000M网络，基本用满了。

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : rxe0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0011 PSN 0xfe34df RKey 0x000758 VAddr 0x007f0e0c4fa000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:35:147:56
 remote address: LID 0000 QPN 0x0011 PSN 0xf07e1a RKey 0x00062c VAddr 0x007f564f9ff000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:35:148:198
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2498.879000 != 2600.320000. CPU Frequency is not max.
 65536      5000             103.59             102.24             0.001636
---------------------------------------------------------------------------------------

4.2 读速度测速

读使用ib_read_bw命令，和写类似，不再赘述。读的效率明显较弱，可能一致性问题的原因？

---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF          Device         : rxe0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0011 PSN 0xa526ed OUT 0x80 RKey 0x0007b0 VAddr 0x007f9485896000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:35:147:56
 remote address: LID 0000 QPN 0x0011 PSN 0x2285c5 OUT 0x80 RKey 0x000658 VAddr 0x007fb5ccfad000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:35:148:198
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2546.240000 != 2600.049000. CPU Frequency is not max.
 65536      1000             31.30              17.66              0.000283
---------------------------------------------------------------------------------------