鲲鹏920集成100G网卡RDMA测试说明

已于 2024-06-05 22:08:15 修改
阅读量2.6k 收藏 14
点赞数 3
分类专栏： Felven在职场 文章标签： 网络 rdma 鲲鹏920 华为 rocev2
于 2024-03-17 21:37:16 首次发布
本文为博主原创文章，未经博主允许不得转载。
本文链接：https://blog.csdn.net/zhaoxinfan/article/details/136790348
版权

1、背景介绍

目前鲲鹏920处理器内集成了两个100G网卡，支持RDMA（Roce V2），说明如下

为了测试网卡性能，需要进行RDMA测试，两块鲲鹏920的板卡通过盛科的8180 100G交换芯片实现交换功能。

盛科8180芯片介绍如下：

2、测试步骤

两块KP920模块上均安装统信服务器操作系统V20。

启动后加载rdma驱动

modprobe hns3
modprobe hclge
modprobe hclgevf
modprobe hns_roce_hw_v2

查看网卡信息

[root@localhost Desktop]# ibv_devinfo -d hns_0
hca_id: hns_0
        transport:                      InfiniBand (0)
        fw_ver:                         0.277.001
        node_guid:                      a270:b7ff:fe55:b4c6
        sys_image_guid:                 a270:b7ff:fe55:b4c6
        vendor_id:                      0x19e5
        vendor_part_id:                 41510
        hw_ver:                         0x100
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

系统内默认已经安装了rdma-core和perftest，无需单独安装。

查看网卡状态，输入

rdma link
[root@localhost Desktop]# rdma link
link hns_0/1 state ACTIVE physical_state LINK_UP netdev enp125s0f0 
link hns_1/1 state DOWN physical_state DISABLED netdev enp125s0f1

然后再使用perftest内部集成的八种工具进行RDMA性能测试

  • ib_atomic_bw
  • ib_read_bw
  • ib_send_bw
  • ib_write_bw
  • ib_atomic_lat
  • ib_read_lat
  • ib_send_lat
  • ib_write_lat

这些测试工具分别用来测试不同的操作类型的时延和带宽性能。

详细操作步骤可以参考这篇文档：WayCa-Kunpeng-底软/Wayca-Kunpeng-高速网络/WayCa-Kunpeng-高速网络-RoCE用户使用指南.md · openEuler/WayCa - Gitee.com

3、iperf测试结果

首先Iperf四线程测试速率为97.9Gbps，五线程为98.7Gbps：

[root@localhost home]# taskset -c 16-23 iperf -s -p 7000
------------------------------------------------------------
Server listening on TCP port 7000
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 37070
[  5] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 37072
[  8] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 37074
[  7] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 37076
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-15.8 sec  45.0 GBytes  24.4 Gbits/sec
[  5]  0.0-15.8 sec  44.3 GBytes  24.0 Gbits/sec
[  8]  0.0-15.8 sec  45.5 GBytes  24.7 Gbits/sec
[  7]  0.0-15.8 sec  45.5 GBytes  24.7 Gbits/sec
[SUM]  0.0-15.8 sec   180 GBytes  97.9 Gbits/sec
[  9] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 49844
[  4] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 49846
[  5] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 49840
[  7] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 49842
[ 10] local 192.170.10.10 port 7000 connected with 192.170.10.20 port 49848
[  9]  0.0-21.2 sec  50.9 GBytes  20.6 Gbits/sec
[  4]  0.0-21.2 sec  48.5 GBytes  19.6 Gbits/sec
[  5]  0.0-21.2 sec  47.5 GBytes  19.2 Gbits/sec
[  7]  0.0-21.2 sec  46.8 GBytes  18.9 Gbits/sec
[ 10]  0.0-21.2 sec  50.3 GBytes  20.3 Gbits/sec
[SUM]  0.0-21.2 sec   244 GBytes  98.7 Gbits/sec
[root@localhost Desktop]# taskset -c 16-23 iperf -c 192.170.10.10 -p 7000 -t 60 -i 10 -P 4
------------------------------------------------------------
Client connecting to 192.170.10.10, TCP port 7000
TCP window size: 3.14 MByte (default)
------------------------------------------------------------
[  3] local 192.170.10.20 port 37072 connected with 192.170.10.10 port 7000
[  5] local 192.170.10.20 port 37076 connected with 192.170.10.10 port 7000
[  4] local 192.170.10.20 port 37070 connected with 192.170.10.10 port 7000
[  6] local 192.170.10.20 port 37074 connected with 192.170.10.10 port 7000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  28.0 GBytes  24.0 Gbits/sec
[  5]  0.0-10.0 sec  28.8 GBytes  24.7 Gbits/sec
[  4]  0.0-10.0 sec  28.4 GBytes  24.4 Gbits/sec
[  6]  0.0-10.0 sec  28.8 GBytes  24.8 Gbits/sec
[SUM]  0.0-10.0 sec   114 GBytes  97.8 Gbits/sec
^C[  3]  0.0-15.8 sec  44.3 GBytes  24.0 Gbits/sec
[  5]  0.0-15.8 sec  45.5 GBytes  24.7 Gbits/sec
[  4]  0.0-15.8 sec  45.0 GBytes  24.4 Gbits/sec
[  6]  0.0-15.8 sec  45.5 GBytes  24.7 Gbits/sec
[SUM]  0.0-15.8 sec   180 GBytes  97.9 Gbits/sec
[root@localhost Desktop]# taskset -c 16-23 iperf -c 192.170.10.10 -p 7000 -t 60 -i 10 -P 5
------------------------------------------------------------
Client connecting to 192.170.10.10, TCP port 7000
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[  3] local 192.170.10.20 port 49842 connected with 192.170.10.10 port 7000
[  7] local 192.170.10.20 port 49846 connected with 192.170.10.10 port 7000
[  4] local 192.170.10.20 port 49840 connected with 192.170.10.10 port 7000
[  6] local 192.170.10.20 port 49844 connected with 192.170.10.10 port 7000
[  5] local 192.170.10.20 port 49848 connected with 192.170.10.10 port 7000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  22.5 GBytes  19.4 Gbits/sec
[  7]  0.0-10.0 sec  22.7 GBytes  19.5 Gbits/sec
[  4]  0.0-10.0 sec  21.2 GBytes  18.2 Gbits/sec
[  6]  0.0-10.0 sec  24.1 GBytes  20.7 Gbits/sec
[  5]  0.0-10.0 sec  24.4 GBytes  21.0 Gbits/sec
[SUM]  0.0-10.0 sec   115 GBytes  98.7 Gbits/sec
[  3] 10.0-20.0 sec  21.9 GBytes  18.8 Gbits/sec
[  7] 10.0-20.0 sec  22.9 GBytes  19.6 Gbits/sec
[  4] 10.0-20.0 sec  23.7 GBytes  20.4 Gbits/sec
[  6] 10.0-20.0 sec  23.6 GBytes  20.3 Gbits/sec
[  5] 10.0-20.0 sec  22.8 GBytes  19.6 Gbits/sec
[SUM] 10.0-20.0 sec   115 GBytes  98.7 Gbits/sec
^C[  3]  0.0-21.2 sec  46.8 GBytes  18.9 Gbits/sec
[  7]  0.0-21.2 sec  48.5 GBytes  19.6 Gbits/sec
[  4]  0.0-21.2 sec  47.5 GBytes  19.2 Gbits/sec
[  6]  0.0-21.2 sec  50.9 GBytes  20.6 Gbits/sec
[  5]  0.0-21.2 sec  50.3 GBytes  20.3 Gbits/sec
[SUM]  0.0-21.2 sec   244 GBytes  98.7 Gbits/sec

4、RDMA速率测试

send速率

[root@localhost ~]# ib_send_bw -d hns_1 192.170.1.10 -a --report_gbit
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000d PSN 0x98dc41
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
 remote address: LID 0000 QPN 0x000d PSN 0x820cd5
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          1000           0.061539            0.058761            3.672573
 4          1000             0.12               0.12               3.771477
 8          1000             0.25               0.24               3.783579
 16         1000             0.49               0.48               3.786343
 32         1000             0.98               0.96               3.767650
 64         1000             1.97               1.94               3.785453
 128        1000             3.79               3.79               3.698363
 256        1000             7.88               7.73               3.772159
 512        1000             15.75              15.43              3.767074
 1024       1000             31.51              30.86              3.767186
 2048       1000             63.02              61.68              3.764813
 4096       1000             88.56              88.52              2.701550
 8192       1000             91.02              89.92              1.372036
 16384      1000             91.02              90.65              0.691642
 32768      1000             91.34              91.03              0.347238
 65536      1000             91.18              91.10              0.173766
 131072     1000             91.34              91.29              0.087064
 262144     1000             91.30              91.30              0.043534
 524288     1000             91.38              91.37              0.021784
 1048576    1000             91.40              91.40              0.010896
 2097152    1000             91.41              91.41              0.005448
 4194304    1000             91.42              91.42              0.002724
 8388608    1000             91.42              91.42              0.001362
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_send_bw -d hns_1 -a --report_gbit

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 RX depth        : 512
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000d PSN 0x820cd5
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
 remote address: LID 0000 QPN 0x000d PSN 0x98dc41
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          1000           0.000000            0.060111            3.756964
 4          1000             0.00               0.12               3.847526
 8          1000             0.00               0.25               3.856431
 16         1000             0.00               0.49               3.864930
 32         1000             0.00               0.99               3.849570
 64         1000             0.00               1.98               3.859243
 128        1000             0.00               3.86               3.768928
 256        1000             0.00               7.89               3.852723
 512        1000             0.00               15.87              3.874202
 1024       1000             0.00               31.57              3.853310
 2048       1000             0.00               63.86              3.897894
 4096       1000             0.00               90.73              2.768949
 8192       1000             0.00               91.11              1.390244
 16384      1000             0.00               91.30              0.696563
 32768      1000             0.00               91.41              0.348711
 65536      1000             0.00               91.37              0.174273
 131072     1000             0.00               91.49              0.087253
 262144     1000             0.00               91.48              0.043619
 524288     1000             0.00               91.51              0.021817
 1048576    1000             0.00               91.51              0.010908
 2097152    1000             0.00               91.51              0.005454
 4194304    1000             0.00               91.51              0.002727
 8388608    1000             0.00               91.51              0.001364
---------------------------------------------------------------------------------------

write速率

[root@localhost ~]# ib_write_bw -d hns_1 -a --report_gbit

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000c PSN 0x328545 RKey 0x000300 VAddr 0x00ffffa1610000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
 remote address: LID 0000 QPN 0x000c PSN 0x5f384d RKey 0x000300 VAddr 0x00ffff82740000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8388608    5000             91.42              91.42              0.001362
---------------------------------------------------------------------------------------

[root@localhost ~]# ib_write_bw -d hns_1 192.170.1.10 -a --report_gbit
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000c PSN 0x5f384d RKey 0x000300 VAddr 0x00ffff82740000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
 remote address: LID 0000 QPN 0x000c PSN 0x328545 RKey 0x000300 VAddr 0x00ffffa1610000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          5000           0.076191            0.072592            4.537006
 4          5000             0.15               0.15               4.693742
 8          5000             0.30               0.30               4.694841
 16         5000             0.61               0.60               4.697291
 32         5000             1.22               1.20               4.695094
 64         5000             2.44               2.40               4.696643
 128        5000             4.88               4.71               4.597502
 256        5000             9.75               9.52               4.648410
 512        5000             19.50              19.05              4.649918
 1024       5000             39.01              38.14              4.655201
 2048       5000             78.02              75.80              4.626161
 4096       5000             93.62              91.08              2.779397
 8192       5000             92.30              91.32              1.393386
 16384      5000             91.66              91.37              0.697134
 32768      5000             91.66              91.39              0.348636
 65536      5000             91.50              91.40              0.174337
 131072     5000             91.42              91.41              0.087176
 262144     5000             91.42              91.42              0.043591
 524288     5000             91.44              91.42              0.021796
 1048576    5000             91.43              91.42              0.010898
 2097152    5000             91.43              91.42              0.005449
 4194304    5000             91.42              91.42              0.002725
 8388608    5000             91.42              91.42              0.001362
---------------------------------------------------------------------------------------

read速率

[root@localhost ~]# ib_read_bw -d hns_1 192.170.1.10 -a --report_gbit
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000b PSN 0x396c6d OUT 0x80 RKey 0x000300 VAddr 0x00ffffa9e90000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
 remote address: LID 0000 QPN 0x000b PSN 0x613319 OUT 0x80 RKey 0x000300 VAddr 0x00ffff9aa80000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          1000           0.061539            0.058837            3.677329
 4          1000             0.12               0.12               3.789605
 8          1000             0.25               0.24               3.791202
 16         1000             0.49               0.49               3.789773
 32         1000             0.98               0.97               3.797229
 64         1000             1.97               1.94               3.793508
 128        1000             3.94               3.88               3.785345
 256        1000             7.88               7.74               3.777609
 512        1000             15.75              15.44              3.770211
 1024       1000             31.51              30.89              3.771180
 2048       1000             63.02              61.86              3.775333
 4096       1000             91.02              90.38              2.758042
 8192       1000             91.02              90.91              1.387160
 16384      1000             91.66              91.15              0.695423
 32768      1000             91.34              91.26              0.348113
 65536      1000             91.34              91.32              0.174179
 131072     1000             91.42              91.35              0.087121
 262144     1000             91.42              91.38              0.043574
 524288     1000             91.42              91.41              0.021793
 1048576    1000             91.42              91.41              0.010897
 2097152    1000             91.42              91.42              0.005449
 4194304    1000             91.42              91.42              0.002725
 8388608    1000             91.42              91.42              0.001362
---------------------------------------------------------------------------------------

5、RDMA时延测试

send时延

[root@localhost ~]# ib_send_lat -d hns_1 -a --report_gbit

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 RX depth        : 512
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000e PSN 0x62ffca
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
 remote address: LID 0000 QPN 0x000e PSN 0xb72844
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          2.15           2.65         2.25                2.26             0.06            2.43                    2.65
 4       1000          2.16           2.84         2.23                2.24             0.05            2.41                    2.84
 8       1000          2.16           2.74         2.23                2.23             0.04            2.40                    2.74
 16      1000          2.16           2.46         2.23                2.24             0.04            2.33                    2.46
 32      1000          2.16           3.06         2.25                2.25             0.05            2.42                    3.06
 64      1000          2.24           2.86         2.31                2.32             0.04            2.45                    2.86
 128     1000          2.26           2.59         2.32                2.33             0.04            2.47                    2.59
 256     1000          2.29           2.83         2.35                2.36             0.04            2.49                    2.83
 512     1000          2.37           3.60         2.45                2.45             0.07            2.69                    3.60
 1024    1000          2.45           2.79         2.52                2.53             0.04            2.67                    2.79
 2048    1000          2.55           4.28         2.61                2.62             0.07            2.76                    4.28
 4096    1000          2.74           4.63         2.82                2.83             0.09            3.04                    4.63
 8192    1000          3.13           5.19         3.49                3.51             0.09            3.72                    5.19
 16384   1000          4.47           6.57         4.89                4.88             0.11            5.13                    6.57
 32768   1000          7.33           9.57         7.67                7.70             0.11            7.96                    9.57
 65536   1000          12.74          14.45        13.11               13.14            0.13            13.42                   14.45
 131072  1000          18.66          20.80        18.85               18.87            0.13            19.15                   20.80
 262144  1000          30.15          32.51        30.38               30.39            0.14            30.68                   32.51
 524288  1000          53.07          55.18        53.28               53.29            0.13            53.53                   55.18
 1048576 1000          98.96          100.85       99.25               99.26            0.13            99.58                   100.85
 2097152 1000          190.76         192.65       190.99              191.01           0.15            191.38                  192.65
 4194304 1000          374.18         376.28       374.55              374.56           0.15            374.92                  376.28
 8388608 1000          741.17         743.33       741.57              741.58           0.15            741.93                  743.33
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_send_lat -d hns_1 192.170.1.10 -a --report_gbit
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000e PSN 0xb72844
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
 remote address: LID 0000 QPN 0x000e PSN 0x62ffca
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          2.15           2.71         2.25                2.26             0.06            2.45                    2.71
 4       1000          2.16           2.85         2.23                2.24             0.05            2.42                    2.85
 8       1000          2.16           2.74         2.23                2.23             0.04            2.41                    2.74
 16      1000          2.16           2.63         2.23                2.24             0.04            2.33                    2.63
 32      1000          2.17           4.46         2.25                2.25             0.05            2.42                    4.46
 64      1000          2.25           2.88         2.31                2.32             0.04            2.45                    2.88
 128     1000          2.25           2.76         2.32                2.33             0.04            2.50                    2.76
 256     1000          2.28           2.81         2.35                2.36             0.04            2.49                    2.81
 512     1000          2.37           5.07         2.44                2.45             0.07            2.65                    5.07
 1024    1000          2.45           2.94         2.52                2.53             0.05            2.67                    2.94
 2048    1000          2.54           5.93         2.61                2.62             0.05            2.75                    5.93
 4096    1000          2.74           6.17         2.81                2.83             0.09            3.16                    6.17
 8192    1000          3.12           6.84         3.47                3.51             0.11            3.84                    6.84
 16384   1000          4.17           8.50         4.81                4.88             0.15            5.22                    8.50
 32768   1000          7.18           10.98        7.74                7.70             0.19            8.09                    10.98
 65536   1000          12.47          15.52        13.10               13.13            0.14            13.55                   15.52
 131072  1000          18.66          22.42        18.83               18.87            0.14            19.29                   22.42
 262144  1000          30.13          34.86        30.37               30.39            0.17            30.76                   34.86
 524288  1000          53.08          56.96        53.27               53.29            0.12            53.57                   56.96
 1048576 1000          98.91          102.53       99.27               99.26            0.12            99.59                   102.53
 2097152 1000          190.72         194.48       190.99              191.01           0.14            191.39                  194.48
 4194304 1000          374.18         378.00       374.56              374.56           0.15            374.90                  378.00
 8388608 1000          741.20         744.99       741.57              741.58           0.15            741.96                  744.99
---------------------------------------------------------------------------------------

write时延

[root@localhost ~]# ib_write_lat -d hns_1 -a --report_gbit

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000f PSN 0x6f40ee RKey 0x000300 VAddr 0x00ffff8a5b0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
 remote address: LID 0000 QPN 0x000f PSN 0x380ec3 RKey 0x000300 VAddr 0x00ffff99a00000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          2.00           2.14         2.04                2.05             0.02            2.09                    2.14
 4       1000          1.98           2.27         2.02                2.03             0.01            2.06                    2.27
 8       1000          1.98           2.91         2.02                2.02             0.02            2.05                    2.91
 16      1000          1.99           2.38         2.03                2.03             0.01            2.05                    2.38
 32      1000          1.99           2.13         2.03                2.03             0.01            2.06                    2.13
 64      1000          2.07           2.36         2.10                2.11             0.01            2.15                    2.36
 128     1000          2.08           2.21         2.10                2.10             0.01            2.12                    2.21
 256     1000          2.10           2.16         2.12                2.12             0.01            2.14                    2.16
 512     1000          2.17           2.26         2.18                2.19             0.01            2.20                    2.26
 1024    1000          2.26           2.54         2.28                2.28             0.02            2.36                    2.54
 2048    1000          2.39           3.92         2.40                2.41             0.01            2.44                    3.92
 4096    1000          2.68           3.33         2.70                2.72             0.08            3.02                    3.33
 8192    1000          3.12           3.76         3.42                3.46             0.10            3.73                    3.76
 16384   1000          3.98           5.13         4.78                4.84             0.14            5.10                    5.13
 32768   1000          7.31           8.98         7.67                7.63             0.16            7.93                    8.98
 65536   1000          10.43          13.38        12.95               12.99            0.14            13.36                   13.38
 131072  1000          18.61          19.55        18.71               18.75            0.11            19.10                   19.55
 262144  1000          30.08          30.57        30.18               30.22            0.11            30.55                   30.57
 524288  1000          53.00          53.68        53.16               53.19            0.12            53.51                   53.68
 1048576 1000          98.84          99.49        99.04               99.07            0.12            99.44                   99.49
 2097152 1000          190.47         191.16       190.82              190.78           0.11            191.06                  191.16
 4194304 1000          373.90         374.58       374.12              374.16           0.12            374.52                  374.58
 8388608 1000          740.83         741.55       741.01              741.06           0.11            741.44                  741.55
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_write_lat -d hns_1 192.170.1.10 -a --report_gbit
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000f PSN 0x380ec3 RKey 0x000300 VAddr 0x00ffff99a00000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
 remote address: LID 0000 QPN 0x000f PSN 0x6f40ee RKey 0x000300 VAddr 0x00ffff8a5b0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          2.00           2.14         2.04                2.05             0.02            2.09                    2.14
 4       1000          1.98           2.17         2.02                2.03             0.01            2.06                    2.17
 8       1000          1.98           2.93         2.02                2.02             0.01            2.05                    2.93
 16      1000          1.99           2.11         2.03                2.03             0.01            2.05                    2.11
 32      1000          1.99           2.10         2.03                2.03             0.01            2.07                    2.10
 64      1000          2.06           2.20         2.10                2.11             0.01            2.14                    2.20
 128     1000          2.08           2.17         2.10                2.10             0.01            2.13                    2.17
 256     1000          2.10           2.15         2.12                2.12             0.01            2.14                    2.15
 512     1000          2.17           2.23         2.18                2.19             0.01            2.20                    2.23
 1024    1000          2.26           2.53         2.28                2.28             0.02            2.36                    2.53
 2048    1000          2.39           2.73         2.40                2.41             0.01            2.44                    2.73
 4096    1000          2.67           2.89         2.70                2.72             0.05            2.87                    2.89
 8192    1000          3.27           3.65         3.42                3.46             0.07            3.60                    3.65
 16384   1000          4.50           5.04         4.88                4.84             0.08            4.99                    5.04
 32768   1000          7.47           8.91         7.61                7.63             0.09            7.83                    8.91
 65536   1000          11.68          13.37        12.96               12.99            0.10            13.23                   13.37
 131072  1000          18.60          19.78        18.73               18.75            0.09            19.02                   19.78
 262144  1000          30.07          30.57        30.20               30.22            0.09            30.51                   30.57
 524288  1000          53.03          53.71        53.19               53.19            0.09            53.43                   53.71
 1048576 1000          98.87          99.36        99.07               99.07            0.10            99.32                   99.36
 2097152 1000          190.47         191.18       190.81              190.78           0.11            191.07                  191.18
 4194304 1000          373.90         374.62       374.13              374.16           0.12            374.52                  374.62
 8388608 1000          740.82         741.54       741.02              741.06           0.11            741.41                  741.54
---------------------------------------------------------------------------------------

read时延

[root@localhost ~]# ib_read_lat -d hns_1 -a --report_gbit
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0010 PSN 0x268850 OUT 0x80 RKey 0x000300 VAddr 0x00ffff9c380000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
 remote address: LID 0000 QPN 0x0010 PSN 0x56ebb OUT 0x80 RKey 0x000300 VAddr 0x00ffff8fb80000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_read_lat -d hns_1 192.170.1.10 -a --report_gbit
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 2
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0010 PSN 0x56ebb OUT 0x80 RKey 0x000300 VAddr 0x00ffff8fb80000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
 remote address: LID 0000 QPN 0x0010 PSN 0x268850 OUT 0x80 RKey 0x000300 VAddr 0x00ffff9c380000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:10
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          3.37           5.71         3.43                3.43             0.04            3.58                    5.71
 4       1000          3.36           3.86         3.43                3.43             0.04            3.59                    3.86
 8       1000          3.37           3.60         3.43                3.43             0.03            3.57                    3.60
 16      1000          3.36           3.62         3.44                3.43             0.03            3.58                    3.62
 32      1000          3.38           3.77         3.45                3.45             0.03            3.60                    3.77
 64      1000          3.45           4.06         3.51                3.51             0.04            3.69                    4.06
 128     1000          3.43           3.77         3.50                3.50             0.03            3.66                    3.77
 256     1000          3.46           3.76         3.53                3.53             0.03            3.69                    3.76
 512     1000          3.54           3.82         3.59                3.60             0.04            3.81                    3.82
 1024    1000          3.63           3.85         3.67                3.67             0.02            3.82                    3.85
 2048    1000          3.73           4.27         3.78                3.79             0.04            3.95                    4.27
 4096    1000          3.91           4.35         3.95                3.98             0.08            4.28                    4.35
 8192    1000          4.56           5.08         4.62                4.66             0.10            4.93                    5.08
 16384   1000          5.59           6.40         5.94                6.01             0.13            6.25                    6.40
 32768   1000          8.40           9.25         8.82                8.71             0.15            9.00                    9.25
 65536   1000          11.70          14.96        14.26               14.31            0.16            14.84                   14.96
 131072  1000          19.89          20.91        20.04               20.13            0.19            20.68                   20.91
 262144  1000          31.51          32.91        31.67               31.75            0.20            32.34                   32.91
 524288  1000          54.43          55.32        54.56               54.63            0.17            55.16                   55.32
 1048576 1000          100.35         101.73       100.50              100.58           0.19            101.23                  101.73
 2097152 1000          192.10         193.21       192.25              192.34           0.21            193.06                  193.21
 4194304 1000          375.62         376.71       375.87              375.89           0.19            376.43                  376.71
 8388608 1000          742.64         743.73       742.82              742.91           0.21            743.48                  743.73
---------------------------------------------------------------------------------------

上面MTU值为1024，当MTU修改为4096后测试结果如下：

[root@localhost ~]# ib_send_bw -d hns_1 192.170.1.20 -a --report_gbits
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0008 PSN 0x81ed9e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:30
 remote address: LID 0000 QPN 0x0008 PSN 0xbc71f4
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          1000           0.072728            0.069845            4.365336
 4          1000             0.15               0.14               4.516115
 8          1000             0.29               0.29               4.524693
 16         1000             0.58               0.58               4.529606
 32         1000             1.16               1.16               4.524926
 64         1000             2.33               2.32               4.527814
 128        1000             4.65               4.63               4.519391
 256        1000             9.31               9.22               4.502882
 512        1000             18.62              18.41              4.493801
 1024       1000             37.24              36.96              4.511237
 2048       1000             74.47              73.14              4.463917
 4096       1000             96.38              93.84              2.863877
 8192       1000             96.38              95.70              1.460269
 16384      1000             97.09              96.65              0.737381
 32768      1000             97.09              97.05              0.370210
 65536      1000             97.45              97.28              0.185551
 131072     1000             97.36              97.35              0.092844
 262144     1000             97.41              97.38              0.046433
 524288     1000             97.50              97.49              0.023243
 1048576    1000             97.52              97.52              0.011625
 2097152    1000             97.54              97.54              0.005814
 4194304    1000             97.55              97.55              0.002907
 8388608    1000             97.55              97.55              0.001454
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_read_bw -d hns_1 192.170.1.20 -a --report_gbits
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0009 PSN 0xa96e OUT 0x80 RKey 0x000300 VAddr 0x00ffff8cfb0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:30
 remote address: LID 0000 QPN 0x0009 PSN 0x627b4d OUT 0x80 RKey 0x000300 VAddr 0x00ffffbd1c0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          1000           0.072727            0.069475            4.342159
 4          1000             0.15               0.14               4.496640
 8          1000             0.29               0.28               4.441089
 16         1000             0.58               0.58               4.509786
 32         1000             1.16               1.15               4.500889
 64         1000             2.33               2.31               4.507341
 128        1000             4.65               4.63               4.520441
 256        1000             9.31               9.22               4.501272
 512        1000             18.62              18.39              4.490803
 1024       1000             37.24              36.84              4.497052
 2048       1000             74.47              73.29              4.473521
 4096       1000             96.38              96.31              2.939221
 8192       1000             97.81              96.89              1.478349
 16384      1000             97.82              97.21              0.741637
 32768      1000             97.45              97.36              0.371409
 65536      1000             97.45              97.43              0.185837
 131072     1000             97.54              97.47              0.092958
 262144     1000             97.54              97.51              0.046495
 524288     1000             97.54              97.54              0.023254
 1048576    1000             97.55              97.55              0.011628
 2097152    1000             97.55              97.55              0.005814
 4194304    1000             97.55              97.55              0.002907
 8388608    1000             97.56              97.56              0.001454
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_write_bw -d hns_1 192.170.1.20 -a --report_gbits
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000a PSN 0x930384 RKey 0x000300 VAddr 0x00ffffbd5a0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:30
 remote address: LID 0000 QPN 0x000a PSN 0xd109ee RKey 0x000300 VAddr 0x00ffff81c20000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          5000           0.072727            0.071706            4.481610
 4          5000             0.15               0.15               4.579037
 8          5000             0.30               0.29               4.580150
 16         5000             0.61               0.59               4.578247
 32         5000             1.22               1.17               4.584232
 64         5000             2.44               2.35               4.582672
 128        5000             4.65               4.65               4.538125
 256        5000             9.31               9.30               4.540751
 512        5000             18.62              18.60              4.539880
 1024       5000             37.24              37.23              4.544099
 2048       5000             74.47              73.67              4.496749
 4096       5000             99.30              97.01              2.960643
 8192       5000             97.81              97.39              1.486108
 16384      5000             97.82              97.47              0.743649
 32768      5000             97.81              97.52              0.372008
 65536      5000             97.63              97.53              0.186030
 131072     5000             97.54              97.54              0.093021
 262144     5000             97.59              97.55              0.046515
 524288     5000             97.57              97.55              0.023258
 1048576    5000             97.57              97.55              0.011629
 2097152    5000             97.56              97.55              0.005815
 4194304    5000             97.56              97.56              0.002907
 8388608    5000             97.56              97.56              0.001454
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_write_lat -d hns_1 192.170.1.20 -a --report_gbits
---------------------------------------------------------------------------------------
                    RDMA_Write Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000b PSN 0x40ca6d RKey 0x000300 VAddr 0x00ffffab090000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:30
 remote address: LID 0000 QPN 0x000b PSN 0xb564da RKey 0x000300 VAddr 0x00ffff908e0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          1.92           4.57         1.97                1.98             0.03            2.04                    4.57
 4       1000          1.92           2.14         1.97                1.97             0.01            2.00                    2.14
 8       1000          1.92           2.03         1.97                1.97             0.01            2.00                    2.03
 16      1000          1.94           2.08         1.98                1.98             0.01            2.01                    2.08
 32      1000          1.94           2.05         1.98                1.98             0.01            2.01                    2.05
 64      1000          2.00           2.19         2.04                2.04             0.01            2.07                    2.19
 128     1000          2.02           2.07         2.04                2.04             0.01            2.05                    2.07
 256     1000          2.03           2.16         2.05                2.05             0.01            2.07                    2.16
 512     1000          2.10           2.81         2.12                2.12             0.01            2.14                    2.81
 1024    1000          2.25           2.53         2.27                2.28             0.02            2.38                    2.53
 2048    1000          2.44           2.51         2.45                2.46             0.01            2.47                    2.51
 4096    1000          2.96           3.15         2.98                2.99             0.03            3.14                    3.15
 8192    1000          3.43           3.92         3.70                3.71             0.04            3.86                    3.92
 16384   1000          4.59           5.17         4.99                5.01             0.05            5.15                    5.17
 32768   1000          6.91           7.83         7.52                7.56             0.09            7.79                    7.83
 65536   1000          11.40          13.13        12.68               12.66            0.09            12.81                   13.13
 131072  1000          17.90          18.26        18.06               18.05            0.08            18.24                   18.26
 262144  1000          28.64          29.21        28.80               28.82            0.12            29.12                   29.21
 524288  1000          50.03          50.64        50.33               50.33            0.10            50.60                   50.64
 1048576 1000          93.15          93.56        93.33               93.32            0.09            93.50                   93.56
 2097152 1000          179.12         179.60       179.30              179.30           0.09            179.52                  179.60
 4194304 1000          351.01         351.57       351.28              351.28           0.09            351.50                  351.57
 8388608 1000          694.92         695.62       695.24              695.24           0.12            695.51                  695.62
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_read_lat -d hns_1 192.170.1.20 -a --report_gbits
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000c PSN 0x86634e OUT 0x80 RKey 0x000300 VAddr 0x00ffff81510000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:30
 remote address: LID 0000 QPN 0x000c PSN 0xa7f013 OUT 0x80 RKey 0x000300 VAddr 0x00ffff8e0b0000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          3.24           5.20         3.31                3.31             0.04            3.43                    5.20
 4       1000          3.22           3.68         3.29                3.29             0.03            3.37                    3.68
 8       1000          3.23           3.40         3.29                3.29             0.03            3.36                    3.40
 16      1000          3.23           3.47         3.30                3.30             0.03            3.38                    3.47
 32      1000          3.24           3.48         3.30                3.30             0.03            3.39                    3.48
 64      1000          3.30           3.46         3.37                3.37             0.02            3.45                    3.46
 128     1000          3.32           3.59         3.39                3.39             0.03            3.47                    3.59
 256     1000          3.36           3.75         3.42                3.42             0.03            3.49                    3.75
 512     1000          3.50           3.74         3.54                3.55             0.03            3.72                    3.74
 1024    1000          3.59           3.79         3.64                3.64             0.02            3.76                    3.79
 2048    1000          3.83           4.17         3.87                3.87             0.02            4.00                    4.17
 4096    1000          4.47           4.84         4.51                4.52             0.04            4.81                    4.84
 8192    1000          5.12           5.54         5.20                5.21             0.06            5.50                    5.54
 16384   1000          6.17           6.82         6.48                6.50             0.08            6.79                    6.82
 32768   1000          8.21           9.54         9.01                9.06             0.11            9.40                    9.54
 65536   1000          12.00          14.91        14.24               14.31            0.17            14.77                   14.91
 131072  1000          19.49          20.23        19.62               19.69            0.16            20.15                   20.23
 262144  1000          30.25          31.14        30.43               30.49            0.18            30.97                   31.14
 524288  1000          51.75          52.57        51.89               51.96            0.15            52.36                   52.57
 1048576 1000          94.73          95.45        94.91               94.98            0.16            95.41                   95.45
 2097152 1000          180.74         181.59       181.01              181.02           0.18            181.51                  181.59
 4194304 1000          352.73         353.58       352.96              352.98           0.18            353.50                  353.58
 8388608 1000          696.68         697.43       696.89              696.93           0.18            697.31                  697.43
---------------------------------------------------------------------------------------
[root@localhost ~]# ib_send_lat -d hns_1 192.170.1.20 -a --report_gbits
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF          Device         : hns_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000d PSN 0xfffcfe
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:30
 remote address: LID 0000 QPN 0x000d PSN 0x1e10e8
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:170:01:20
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
 2       1000          2.12           4.73         2.21                2.21             0.09            2.35                    4.73
 4       1000          2.10           2.52         2.18                2.19             0.04            2.28                    2.52
 8       1000          2.11           2.55         2.20                2.20             0.05            2.40                    2.55
 16      1000          2.11           3.77         2.19                2.19             0.04            2.33                    3.77
 32      1000          2.12           3.75         2.19                2.20             0.04            2.30                    3.75
 64      1000          2.19           2.60         2.26                2.27             0.04            2.36                    2.60
 128     1000          2.22           3.88         2.29                2.29             0.04            2.42                    3.88
 256     1000          2.24           3.11         2.32                2.33             0.04            2.42                    3.11
 512     1000          2.38           2.78         2.45                2.47             0.06            2.63                    2.78
 1024    1000          2.48           2.95         2.55                2.55             0.05            2.75                    2.95
 2048    1000          2.65           3.17         2.72                2.73             0.05            2.95                    3.17
 4096    1000          3.18           4.54         3.24                3.26             0.06            3.55                    4.54
 8192    1000          3.60           7.82         3.95                3.97             0.08            4.27                    7.82
 16384   1000          4.33           9.12         5.24                5.26             0.10            5.60                    9.12
 32768   1000          6.93           11.62        7.87                7.91             0.14            8.30                    11.62
 65536   1000          11.96          15.33        12.97               13.03            0.16            13.41                   15.33
 131072  1000          17.94          22.15        18.36               18.41            0.15            18.73                   22.15
 262144  1000          28.79          33.15        29.23               29.26            0.13            29.61                   33.15
 524288  1000          50.35          54.39        50.70               50.72            0.13            51.04                   54.39
 1048576 1000          93.18          97.45        93.79               93.79            0.15            94.15                   97.45
 2097152 1000          179.22         183.49       179.70              179.74           0.17            180.13                  183.49
 4194304 1000          351.12         355.39       351.64              351.65           0.12            351.94                  355.39
 8388608 1000          695.28         699.56       695.60              695.61           0.12            695.91                  699.56
---------------------------------------------------------------------------------------
[root@localhost ~]#

【以太网RDMA网卡（DPU）技术--背景】
li_kin的博客
12-20 1338
RDMA网卡 DPU RoCEv2
【RDMA】RDMA通信测试工具|RDMA信息查询工具
小鱼菜鸟的博客
07-22 7835
目录 RDMA性能测试工具集-perftest perftest安装 RDMA性能测试(测试性能如何) ibv_xxx_pingpong(带宽和往返时间) ib_send_bw/ib_write_bw(带宽) ib_send_lat\ib_write_lat (时延) ceph_perf_msgr_client/ceph_perf...
【教程】检查RDMA网卡状态和测试带宽 | 附测试脚本
最新发布
xfxuezhang.cn
04-15 1367
脚本测试RDMA带宽
RDMA性能测试工具preftest_README
CSDN博客
09-15 3151
preftest（Performance Test）是一组基于uverbs编写的测试工具集，旨在用作性能微基准。这些测试可以用作硬件或软件调优以及功能测试。uverbs是RDMA（Remote Direct Memory Access）的一种编程接口，它提供了底层、高效的编程接口，可以控制RDMA网络适配器中的硬件资源、创建和管理RDMA操作队列等功能，同时verbs接口也提供了一些原子操作、随机访问、远程直接内存访问等工具，从而实现了高效的、无锁的、直接内存访问。该测试工具集包含了带宽和延迟。
CentOS 7下CX5-RDMA网络测试
一名运维开发工程师的日常记录
09-13 2165
RDMA(Remote Direct Memory Access) 全称远程直接数据存取，就是为了解决网络传输中服务器端数据处理的延迟而产生的。RDMA 通过网络把资料直接传入计算机的存储区，将数据从一个系统快速移动到远程系统存储器中，而不对操作系统造成任何影响，这样就不需要用到多少计算机的处理功能。它消除了外部存储器复制和上下文切换的开销，因而能解放内存带宽和 CPU 周期用于改进应用系统性能。RDMA需要智能网卡支持，这里使用的是Mellanox cx5。基于CentOS 7.8 x86_64。
鲲鹏920 RDMA性能测试
无知的我
12-07 1480
软件说明：统信操作系统，内核版本4.19.90-2305.1.0.0199.56.9.uel20.aarch64。由于当前光模块只有10Gb，速率link为10G。当前的鲲鹏920服务器配置为自带4个25G网卡。硬件设备：两台鲲鹏920服务器。其中只有两个网卡支持RDMA。write带宽性能测试。read带宽性能测试。
iperf3 测试100G网卡带宽性能
热门推荐
xiaosheng124的博客
11-27 3万+
iperf3 iperf3 作为iperf 系列网络测试工具新一代工具，开发团队重写代码使之有全新的实现方式，更少的代码量，更加小巧，但这也导致了其与iperf工具前后不兼容，一些命令执行具有差异化，而这些差异在大带宽网卡性能测试时则更加明显，现记录整理以供大家参考。 iperf 带宽性能测试 首先简单说一下iperf性能测试的命令，由于它是C/S 架构，测试时需要先执行Server端命令： iperf -s -i 5 (-s 是开启server端 -i 5是每5s打印一次数据，可以不加) 之后.
RDMABenchMark:RDMA BenchMark 包括带宽和延迟测试 与传统网络相比较
04-28
**RDMABenchMark：RDMA性能基准测试与传统网络对比** 远程直接内存访问（Remote Direct Memory Access，RDMA）是一种高效的数据传输技术，它允许网络设备直接读写远程计算机的内存，而无需通过操作系统内核。这显著...
Linux查看网卡是rdma,Chelsio RDMA Block设备驱动测试
weixin_36192634的博客
05-01 1607
RDMA Block 设备驱动介绍我们现在测试使用的 Chelsio T4 及 T5 系列万兆以太网卡支持 iWARP RDMA 功能，查阅文档之后发现此系列的万兆网卡除了支持常用的基于 IP 协议的 iSCSI 以及 NFS-RDMA 之类的功能，Chelsio 还特别提供了基于 RDMA 的 Block 设备驱动(以下简称 RBD 驱动)。RDMA 技术本身我就不做详细介绍了，Chelsio ...
linux rdma测试,硬件RDMA的驱动配置和测试
weixin_33262687的博客
05-14 1551
author：headsen chendate： 2019-01-18 10:22:20notice：created by headsen chen himself and not allowed to copy, or you will count law question!版本环境：centos6.8 ，64位，内核：2.6.321，配置网卡：在新卡装上机器，接收光纤，两根线都有接，而且是...
【RDMA】查主机是否支持RDMA|检查可用的RDMA网卡|测试RDMA通信
小鱼菜鸟的博客
04-17 2038
目录 验证RDMA内核模块是否已加载 检查主机上是否有可用的RDMA网卡 测试RDMA通信 原文： https://blog.csdn.net/u012124304/article/details/100543216 验证RDMA内核模块是否已加载 在centos7中，可以用以下命令查看RDMA内核模块的状态： [root@pow...
intel E810 RDMA驱动安装
10-06
本文档详细记录了intel E810 RDMA驱动的安装过程，包括安装前的准备工作、安装RDMA驱动、安装LAN驱动、安装irdma驱动、安装rdma-core库、设置网卡驱动加载模式等步骤。 安装前的准备工作 在安装intel E810 RDMA...
服务器性能测试之Melloanox卡RDMA测试
qq_41897488的博客
11-16 3523
接好外插mellanox网卡并配置IP 客户端： vim /etc/sysconfig/network-scripts/ifcfg-eth0 IPADDR=192.168.64.20 NETMASK=255.255.255.0 NETWORK=192.168.64.0 服务端： vim /etc/sysconfig/network-scripts/ifcfg-eth0 IPADDR=192.168.64.21 NETMASK=255.255.255.0 NETWORK=192.168.64.0 配置好IP后
锐文智能网卡xNIC-2000(100G)在国产鲲鹏服务器测试性能报告
Raymax18的博客
08-19 1505
锐文科技的xNIC系列智能网卡为自主研发，具有自主知识产权的国产智能网卡。通过对国产CPU鲲鹏的适配，在华为TaiShan 2280 v2鲲鹏服务器测试数据如下。 图1:基于鲲鹏CPU服务器的测试平台 鲲鹏CPU（ARM）服务器硬件配置： 服务器 192.168.xxx.xxx 操作系统...
【RDMA】RDMA信息状态查询|网卡安装|诊断工具|测试RDMA网卡是否正常工作InfiniBand IB常用命令|历史命令记录
bandaoyu的note
04-17 2万+
目录 1、常规 IB 监视命令 2、常规 IB 性能测试命令 3、光纤网络诊断工具 4、查询并报告非零 IB 端口计数器 原文;https://support.huawei.com/enterprise/zh/knowledge/EKB1001978581 1、常规 IB 监视命令 ibv_asyncwatch 监视 InfiniBand 异步事件 ibv_devices or ibv_devinfo 列举 InfiniBand 设备或设备信息 ibv_rc_pingpong、ibv_.
iperf3跑满100G网卡实测记录
jacicson1987的专栏
07-27 5032
其中发送速率12016406，基本稳定在1200000 KB/s以上。三个窗口分别同时开3个iperf3 client，跑10分钟。三个iperf3 Client, 合并速率大约 92GB/s。Kbps Mbps Gbps换算倍数是 1000。三个窗口分别开3个iperf3 server。ubuntu20.04, 系统设置默认状态。Kpbs 与 KB/s换算关系是 x8。两台服务器 100G网卡对插直连。
验证Linux主机是否支持RDMA
祈晴小义
09-04 9618
文章目录验证RDMA内核模块是否已加载检查主机上是否有可用的RDMA网卡测试RDMA通信 验证RDMA内核模块是否已加载 在centos7中，可以用以下命令查看RDMA内核模块的状态： [root@power27 docs]# /etc/init.d/openibd status HCA driver loaded Configured IPoIB devices: ib0 Curren...
一文告诉你perftest是什么？perftest怎么用来测试RDMA网络设备
嵌入式技术开发
08-05 2867
perftest 是一个用于性能测试的工具集，专门针对 RDMA（Remote Direct Memory Access）和 InfiniBand 网络进行基准测试。它提供了多种测试程序，允许用户测量不同 RDMA 操作的性能，如延迟和带宽。这些测试通常用于评估 RDMA 设备和网络的性能，帮助用户优化配置或解决潜在问题。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值