本文详细介绍了RDMA(RDMA Remote Direct Memory Access)的测试和诊断工具，包括perftest的安装及性能、功能测试，如ibv_xxx_pingpong、ib_send_bw/ib_write_bw、ibping等，还提到了RDMA内核模块状态的验证方法和报错处理。通过这些工具，可以评估和确保RDMA网络的性能和稳定性。
目录
ceph_perf_msgr_client/ceph_perf_msgr_server
rdma_server, rdma_client (ping-pong测试)
本文作者:bandaoyu @UESTC 不断改进中，请到原文：https://blog.csdn.net/bandaoyu/article/details/115798045
RDMA性能测试工具集-perftest
perftest安装
1、下载源码：
wget https://github.com/linux-rdma/perftest
2、查阅REDME，按指导安装
安装依赖：
yum -y install automake &&yum -y install libtool &&yum -y install pciutils-develcd perftest/
./autogen.sh
./configure Note:If you want to install in a specific directory use the optional flag --prefix=<Directory path> , e.g: ./configure --prefix=<Directory path>
make && make install
sudo ./autogen.sh&&sudo ./configure&&sudo make&&sudo make install报错：
[root@localhost perftest-master]# ./autogen.sh
./autogen.sh: line 5: aclocal: command not found
./autogen.sh: line 6: libtoolize: command not found
./autogen.sh: line 7: autoheader: command not found
./autogen.sh: line 8: automake: command not found
./autogen.sh: line 9: autoconf: command not found
原因：
缺少automake
解决方法：安装
yum -y install automake
报错：libtoolize: command not found
原因：缺少libtool
解决方法：安装
yum -y install libtool
报错：configure: error: pciutils header files not found, consider installing pciutils-devel原因：缺少pciutils-devel
解决方法：安装
yum -y install pciutils-devel
RDMA性能测试(测试性能如何)
RDMA性能测试工具集-perftest
ib_send_lat latency test with send transactions
ib_send_bw bandwidth test with send transactions
ib_write_lat latency test with RDMA write transactions
ib_write_bw bandwidth test with RDMA write transactions
ib_read_lat latency test with RDMA read transactions
ib_read_bw bandwidth test with RDMA read transactions
ib_atomic_lat latency test with atomic transactions
ib_atomic_bw bandwidth test with atomic transactions连接https://github.com/linux-rdma/perftest
注意，性能测试时，注意cpu、内存等是否会成为瓶颈。
ibv_xxx_pingpong(带宽和往返时间)
使用ibv_xxx_pingpong可以测试RDMA设备的流量发送功能：
# 在服务端
ibv_rc_pingpong -g 0 -d mlx4_0 -i 1
local address: LID 0x000c, QPN 0x000a19, PSN 0xf31d1e, GID fe80::e41d:2d03:50:e831
remote address: LID 0x000e, QPN 0x000491, PSN 0xfefc9e, GID fe80::e41d:2d03:50:e801
8192000 bytes in 0.01 seconds = 11821.07 Mbit/sec
1000 iters in 0.01 seconds = 5.54 usec/iter#在客户端 192.168.10.27是服务端的地址
ibv_rc_pingpong -g 0 -d mlx4_0 -i 1 192.168.10.27
local address: LID 0x000e, QPN 0x000491, PSN 0xfefc9e, GID fe80::e41d:2d03:50:e801
remote address: LID 0x000c, QPN 0x000a19, PSN 0xf31d1e, GID fe80::e41d:2d03:50:e831
8192000 bytes in 0.01 seconds = 11797.66 Mbit/sec
1000 iters in 0.01 seconds = 5.55 usec/iter
Syntax
ibv_rc_pingpong [-p TCP_port][-d device][-i IB_port][-s size][-r depth] [-n iters][-l level][-e][-h][IP_address]
where:
-
TCP_port is the TCP port.
-
device is the InfiniBand device.
-
IB_port is the InfiniBand port.
-
size is the size of the ping-pong messages.
-
depth is the number of depth receives to post at one time.
-
iters is the number of message exchanges.
-
level is the service level of the queue pair.
-
IP_address is the IP address of the remote node host.
说明ibv_rc_pingpong Command - Sun Datacenter InfiniBand Switch 648 Topic Set
Options:
-p, --port= listen on/connect to port (default 18515)
-d, --ib-dev= use IB device (default first device found)
-i, --ib-port= use port of IB device (default 1) -s, --size= size of message to exchange (default 4096)
-m, --mtu= path MTU (default 1024)
-r, --rx-depth= number of receives to post at a time (default 500) -n, --iters= number of exchanges (default 1000)
-l, --sl= service level value
-e, --events sleep on CQ events (default poll)
-g, --gid-idx= local port gid index
-c, --contiguous-mr use contiguous mr
-t, --inline-recv= size of inline-recv
-a, --check-nop check NOP opcode
-o, --odp use on demand paging
-u, --upstream use upstream API
-t, --upstream use upstream API
-z, --contig_addr use specifix addr for contig pages MR, must use with -c flag
-b, --ooo enable multipath processing
-j, --memic use device memory
ib_send_bw/ ib_write_bw(带宽)
基本用法：
在A服务器上运行
# ib_send_bw -d rocepxxx # rocepxxx 是A服务器上IP为192.168.5.232的device
在B服务器上运行：
# ib_send_bw -d rocep94s0f0 192.168.5.232 --report_gbits -F #rocep94s0f0是B服务器上IP网段为192.168.5.xxx的device
原文链接：https://blog.csdn.net/bandaoyu/article/details/115791233
ib_send_bw -h查看可知，-a 参数可msg size 递增测试出最大带宽的msg size
[root@localhost ~]# ib_write_bw -R -d iwp175s0f0 -a -F
[root@localhost ~]# ib_write_bw -R -d iwp175s0f0 -i 1 192.169.31.164 -n 1000 -a -F
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
2 30000000 0.00 12.97 6.801028
4 30000000 0.00 25.88 6.784450
8 30000000 0.00 47.67 6.247553
16 30000000 0.00 73.35 4.806851
32 30000000 0.00 144.83 4.745852
64 30000000 0.00 288.74 4.730646
128 30000000 0.00 578.88 4.742147
256 30000000 0.00 1154.31 4.728070
512 30000000 0.00 2303.88 4.718350
1024 30000000 0.00 4336.23 4.440301
2048 30000000 0.00 4390.40 2.247886
4096 30000000 0.00 4409.06 1.128718
……
其他例子：
(mellonx)server、client：
ib_write_bw -d mlx5_0 -i 1 -x 5 --rdma_cm --tos=33 -n 10000000 -s 1M
ib_write_bw 172.17.31.51 -d mlx5_1 -i 1 -x 5 --rdma_cm --tos=33 -n 10000000 -s 1M
(intel)server、client：
[root@localhost ~]# ib_write_bw -R -d iwp175s0f0 -a -F
[root@localhost ~]# ib_write_bw -R -d iwp175s0f0 -i 1 192.169.31.164 -n 1000 -a -F
启用多个QP
ib_write_bw -h
Usage:
ib_write_bw start a server and wait for connection
ib_write_bw connect to server at
Options:
-a, --all Run sizes from 2 till 2^23
-b, --bidirectional 测量双向带宽(默认为单向)
-c, --connection= 连接类型RC / XRC / UC / DC(默认RC)
-d, --ib-dev= 使用IB设备(找到第一个默认设备)
-D, --duration 在自定义的秒数内运行测试。
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq 即使已加载cpufreq_ondemand模块，并且cpu-freq不在最大值，也不会显示警告。
-h, --help Show this
最低0.47元/天 解锁文章
扫一扫
2797
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
