基于 iwarp下的lustre性能测试

近期在做集群网络的升级方案，涉及到网络设备有交换机、线缆、网卡。过程中测试了不少网络厂商的设备，这里主要记录下在测试Chelsio网卡时的一些问题。

测试内容

• 网卡带宽
• 网卡延迟（TCP/iwarp）

• lustre on iwarp

  网卡带宽和延迟这里不做过多介绍，这个测试起来比较简单，直接跑下iperf和ib_write测试工具即可；这里主要记录下如何在lustre上启用iwarp。

  测试环境

  直接上图：
  

存储硬件设备使用DDN/SFA7700x作为lustre的后端盘阵，共60块7.2K NL-SAS硬盘。存储与IO节点之间使用IB互联，IO节点对外使用Chelsio T580-LP-CR 40Gb网卡，客户端有2个，一个是40Gb网卡，一个是10Gb网卡。

环境部署

安装OFED

虽然Mellonax和Chelsio网卡驱动包里都包含了ofed驱动，但如果把两张网卡放在一台机器上使用并加载iwarp功能，ofed驱动会不兼容，这里mellonax厂商可能对其驱动包里的ofed做了修改，阉割了对iwarp的支持。所以我们第一步就是安装开源ofed。

[root@hpc-242-243 ~]# cd OFED-3.18-1
[root@hpc-242-243 OFED-3.18-1]# ls
BUILD_ID  docs  install.pl  LICENSE  ofed.conf.save  README.txt  RPMS  SOURCES  SPECS  SRPMS  uninstall.sh
[root@hpc-242-243 OFED-3.18-1]# ./install.pl 

执行install.pl脚本，根据提示安装即可。安装过程中可能会报错，有些依赖软件需要手动安装

[root@hpc-242-243 ~]#yum install openldap-devel apr-util-devel mod_ssl byacc.x86_64 libtool kernel-devel redhat-rpm-config rpm-build

我的环境里缺的是这些组件。

安装Chelsio网卡驱动

[root@hpc-242-244 ~]# cd ChelsioUwire-2.12.0.2
[root@hpc-242-244 ChelsioUwire-2.12.0.2]# ls
build     dialog.py   docs  install.log  libs    Makefile  rpmbuild     sample_machinefile  specs  support  Uboot
debrules  dialog.pyc  EULA  install.py   lustre  OFED      RPM-Manager  scripts             src    tools    uninstall.log
[root@hpc-242-244 ChelsioUwire-2.12.0.2]# ./install.py -c nic
-------------------------------
Chelsio Unified Installer v2.0 
-------------------------------
Error : The kernel version 2.6.32-504.3.3.el6_lustre.es131.x86_64 is not supported. Refer to README for supported kernel versions.

To compile drivers for an updated kernel, press (Y). To exit, press (N).
Input(Y/N):y
----------------------------------------------------------------------------
| Choice   |  Kernel Version |  Distro                                     |
| Press 1  | 2.6.32-279.el6  | Red Hat Enterprise Linux Server release 6.3 |
| Press 2  | 2.6.32-358.el6  | Red Hat Enterprise Linux Server release 6.4 |
| Press 3  | 2.6.32-431.el6  | Red Hat Enterprise Linux Server release 6.5 |
| Press 4  | 2.6.32-504.el6  | Red Hat Enterprise Linux Server release 6.6 |
| Press 5  | 3.10.0-123.el7  | Red Hat Enterprise Linux Server release 7   |
| Press 6  | 3.10.0-229.el7  | Red Hat Enterprise Linux Server release 7.1 |
| Press 7  | 3.10.0-327.el7  | Red Hat Enterprise Linux Server release 7.2 |
| Press 8  | 2.6.32.12-0.7   | SUSE Linux Enterprise Server 11 SP1         |
| Press 9  | 3.0.13-0.27     | SUSE Linux Enterprise Server 11 SP2         |
| Press 10 | 3.0.76-0.11     | SUSE Linux Enterprise Server 11 SP3         |
| Press 11 | 3.0.101-63      | SUSE Linux Enterprise Server 11 SP4         |
| Press 12 | 3.12.28-4       | SUSE Linux Enterprise Server 12             |
| Press 13 | 3.12.49-11      | SUSE Linux Enterprise Server 12 SP1         |
| Press 14 | 3.13.0-32       | Ubuntu 14.04.1 LTS                          |
| Press 15 | 3.16.0-30       | Ubuntu 14.04.2 LTS                          |
| Press 16 | 3.19.0-25       | Ubuntu 14.04.3 LTS                          |
| Press 17 | 3.4             | Linux kernel Release                        |
| Press 18 | 3.6             | Linux kernel Release                        |
| Press 19 | 3.7             | Linux kernel Release                        |
| Press 20 | 3.8             | Linux kernel Release                        |
| Press 21 | 3.9             | Linux kernel Release                        |
| Press 22 | 3.10            | Linux kernel Release                        |
| Press 23 | 3.11            | Linux kernel Release                        |
| Press 24 | 3.12            | Linux kernel Release                        |
| Press 25 | 3.13            | Linux kernel Release                        |
| Press 26 | 3.14            | Linux kernel Release                        |
| Press 27 | 3.15            | Linux kernel Release                        |
| Press 28 | 3.16            | Linux kernel Release                        |
| Press 29 | 3.17            | Linux kernel Release                        |
| Press 30 | 3.18            | Linux kernel Release                        |
| Press 31 | 4.1             | Linux kernel Release                        |
| Press 32 | exit            |                                             |
----------------------------------------------------------------------------

Input:4
-------------------------------
Chelsio Unified Installer v2.0 
-------------------------------

--------------------------------------------------------------------------------------------
| Choice   |  T4/T5 Configuration               |  Supported Protocols/Drivers             |
--------------------------------------------------------------------------------------------
| Press 1  |  Unified Wire                      |  all the Chelsio drivers                 |
--------------------------------------------------------------------------------------------
| Press 2  |  Low Latency Networks              |  NIC;TOE;RDMA;WD                         |
--------------------------------------------------------------------------------------------
| Press 3  |  High Capacity RDMA                |  NIC;TOE;RDMA                            |
--------------------------------------------------------------------------------------------
| Press 4  |  RDMA Performance                  |  NIC;TOE;RDMA                            |
--------------------------------------------------------------------------------------------
| Press 5  |  High Capacity TOE                 |  NIC;TOE                                 |
--------------------------------------------------------------------------------------------
| Press 6  |  iSCSI Performance                 |  NIC;TOE;iSCSI                           |
--------------------------------------------------------------------------------------------
| Press 7  |  UDP Segmentation Offload & Pacing |  UDP segmenation offload capable NIC;TOE |
--------------------------------------------------------------------------------------------
| Press 8  |  T5 Wire Direct Latency            |  NIC;TOE;RDMA;WD                         |
--------------------------------------------------------------------------------------------
| Press 9  |  T5 High Capacity WD               |  NIC;RDMA;WD                             |
-----------------------------------------