近期在做集群网络的升级方案,涉及到网络设备有交换机、线缆、网卡。过程中测试了不少网络厂商的设备,这里主要记录下在测试Chelsio网卡时的一些问题。
测试内容
- 网卡带宽
- 网卡延迟(TCP/iwarp)
lustre on iwarp
网卡带宽和延迟这里不做过多介绍,这个测试起来比较简单,直接跑下iperf和ib_write测试工具即可;这里主要记录下如何在lustre上启用iwarp。
测试环境
直接上图:
存储硬件设备使用DDN/SFA7700x作为lustre的后端盘阵,共60块7.2K NL-SAS硬盘。存储与IO节点之间使用IB互联,IO节点对外使用Chelsio T580-LP-CR 40Gb网卡,客户端有2个,一个是40Gb网卡,一个是10Gb网卡。
环境部署
安装OFED
虽然Mellonax和Chelsio网卡驱动包里都包含了ofed驱动,但如果把两张网卡放在一台机器上使用并加载iwarp功能,ofed驱动会不兼容,这里mellonax厂商可能对其驱动包里的ofed做了修改,阉割了对iwarp的支持。所以我们第一步就是安装开源ofed。
[root@hpc-242-243 ~]# cd OFED-3.18-1
[root@hpc-242-243 OFED-3.18-1]# ls
BUILD_ID docs install.pl LICENSE ofed.conf.save README.txt RPMS SOURCES SPECS SRPMS uninstall.sh
[root@hpc-242-243 OFED-3.18-1]# ./install.pl
执行install.pl脚本,根据提示安装即可。安装过程中可能会报错,有些依赖软件需要手动安装
[root@hpc-242-243 ~]#yum install openldap-devel apr-util-devel mod_ssl byacc.x86_64 libtool kernel-devel redhat-rpm-config rpm-build
我的环境里缺的是这些组件。
安装Chelsio网卡驱动
[root@hpc-242-244 ~]# cd ChelsioUwire-2.12.0.2
[root@hpc-242-244 ChelsioUwire-2.12.0.2]# ls
build dialog.py docs install.log libs Makefile rpmbuild sample_machinefile specs support Uboot
debrules dialog.pyc EULA install.py lustre OFED RPM-Manager scripts src tools uninstall.log
[root@hpc-242-244 ChelsioUwire-2.12.0.2]# ./install.py -c nic
-------------------------------
Chelsio Unified Installer v2.0
-------------------------------
Error : The kernel version 2.6.32-504.3.3.el6_lustre.es131.x86_64 is not supported. Refer to README for supported kernel versions.
To compile drivers for an updated kernel, press (Y). To exit, press (N).
Input(Y/N):y
----------------------------------------------------------------------------
| Choice | Kernel Version | Distro |
| Press 1 | 2.6.32-279.el6 | Red Hat Enterprise Linux Server release 6.3 |
| Press 2 | 2.6.32-358.el6 | Red Hat Enterprise Linux Server release 6.4 |
| Press 3 | 2.6.32-431.el6 | Red Hat Enterprise Linux Server release 6.5 |
| Press 4 | 2.6.32-504.el6 | Red Hat Enterprise Linux Server release 6.6 |
| Press 5 | 3.10.0-123.el7 | Red Hat Enterprise Linux Server release 7 |
| Press 6 | 3.10.0-229.el7 | Red Hat Enterprise Linux Server release 7.1 |
| Press 7 | 3.10.0-327.el7 | Red Hat Enterprise Linux Server release 7.2 |
| Press 8 | 2.6.32.12-0.7 | SUSE Linux Enterprise Server 11 SP1 |
| Press 9 | 3.0.13-0.27 | SUSE Linux Enterprise Server 11 SP2 |
| Press 10 | 3.0.76-0.11 | SUSE Linux Enterprise Server 11 SP3 |
| Press 11 | 3.0.101-63 | SUSE Linux Enterprise Server 11 SP4 |
| Press 12 | 3.12.28-4 | SUSE Linux Enterprise Server 12 |
| Press 13 | 3.12.49-11 | SUSE Linux Enterprise Server 12 SP1 |
| Press 14 | 3.13.0-32 | Ubuntu 14.04.1 LTS |
| Press 15 | 3.16.0-30 | Ubuntu 14.04.2 LTS |
| Press 16 | 3.19.0-25 | Ubuntu 14.04.3 LTS |
| Press 17 | 3.4 | Linux kernel Release |
| Press 18 | 3.6 | Linux kernel Release |
| Press 19 | 3.7 | Linux kernel Release |
| Press 20 | 3.8 | Linux kernel Release |
| Press 21 | 3.9 | Linux kernel Release |
| Press 22 | 3.10 | Linux kernel Release |
| Press 23 | 3.11 | Linux kernel Release |
| Press 24 | 3.12 | Linux kernel Release |
| Press 25 | 3.13 | Linux kernel Release |
| Press 26 | 3.14 | Linux kernel Release |
| Press 27 | 3.15 | Linux kernel Release |
| Press 28 | 3.16 | Linux kernel Release |
| Press 29 | 3.17 | Linux kernel Release |
| Press 30 | 3.18 | Linux kernel Release |
| Press 31 | 4.1 | Linux kernel Release |
| Press 32 | exit | |
----------------------------------------------------------------------------
Input:4
-------------------------------
Chelsio Unified Installer v2.0
-------------------------------
--------------------------------------------------------------------------------------------
| Choice | T4/T5 Configuration | Supported Protocols/Drivers |
--------------------------------------------------------------------------------------------
| Press 1 | Unified Wire | all the Chelsio drivers |
--------------------------------------------------------------------------------------------
| Press 2 | Low Latency Networks | NIC;TOE;RDMA;WD |
--------------------------------------------------------------------------------------------
| Press 3 | High Capacity RDMA | NIC;TOE;RDMA |
--------------------------------------------------------------------------------------------
| Press 4 | RDMA Performance | NIC;TOE;RDMA |
--------------------------------------------------------------------------------------------
| Press 5 | High Capacity TOE | NIC;TOE |
--------------------------------------------------------------------------------------------
| Press 6 | iSCSI Performance | NIC;TOE;iSCSI |
--------------------------------------------------------------------------------------------
| Press 7 | UDP Segmentation Offload & Pacing | UDP segmenation offload capable NIC;TOE |
--------------------------------------------------------------------------------------------
| Press 8 | T5 Wire Direct Latency | NIC;TOE;RDMA;WD |
--------------------------------------------------------------------------------------------
| Press 9 | T5 High Capacity WD | NIC;RDMA;WD |
-----------------------------------------