【RDMA】intel 因特尔RDMA 驱动和ibverslib 库安装笔记

目录

关键步骤

详细步骤和说明

编译和安装

确认RDMA功能

iWARP / RoCEv2选择

流控制设置

ECN配置

内存需求

资源限制配置文件

RDMA统计

perftest测试工具

动态追踪

动态调试

使用tcpdump捕获RDMA流量

intel 中文官网

报错处理

IP配置

查看device和网口的对应关系

---

关键步骤

(先装ice-->装irdma-->装rdma-core)

1、安装intel RDMA网卡

2、到官网下载驱动https://downloadcenter.intel.com/zh-cn/download/30368/-E810-X722-Ethernet-Linux-RDMA-

3、安装相应的 LAN 驱动程序 (在安装 irdma 之前， E810 和 X722 对应的 LAN 驱动程序(ice 或 i40e)都必须从此版本中包含的源代码构建安装在您的系统上。)

下载ICE包，解压，(看readme，按readme操作)进入src，make install

4、安装irdma

(irdma Linux *驱动程序在支持RDMA的英特尔网络设备上启用RDMA功能。)

5、安装rdma-core (用户空间的ibvers库，给应用程序编程提供接口)

注意，执行 patch -p2 < /path/to/irdma-<version>/libirdma-27.0.patch 命令时，别落了“<”符号

6、设置网卡驱动加载模式iWARP或RoCEv2 

    ibv_devinfo命令查看网卡模式

transport:                      iWARP (1)

详细步骤和说明

原文：readme.txt:https://downloadmirror.intel.com/30368/eng/README_irdma_1.4.22.txt

==============================================================================
irdma - Linux* RDMA Driver for the E810 and X722 Intel(R) Ethernet Controllers
==============================================================================

--------
目录
--------
- Overview                               概览
- Prerequisites                          前提和依赖
- Supported OS List                      支持是操作系统
- Building and Installation              编译和安装
- Confirm RDMA Functionality             配置RDMA功能
- iWARP/RoCEv2 Selection                 选择iWARP或RoCEv2
- iWARP Port Mapper (iwpmd)
- Flow Control Settings
- ECN Configuration
- Devlink Configuration
- Memory Requirements                   内存需求
- Resource Profile Limits               资源限制
- Resource Limits Selector              资源限制选择
- RDMA Statistics                       RDMA统计
- perftest                              性能测试工具
- MPI
- Performance
- Interoperability
- Dynamic Tracing
- Dynamic Debug
- Capturing RDMA Traffic with tcpdump
- Known Issues/Notes

--------
概览
--------

The irdma Linux* driver enables RDMA functionality on RDMA-capable Intel
network devices. Devices supported by this driver:
   - Intel(R) Ethernet Controller E810
   - Intel(R) Ethernet Network Adapter X722

The E810 and X722 devices each support a different set of RDMA features.
    - E810 supports both iWARP and RoCEv2 RDMA transports, and also supports
      congestion management features like priority flow control (PFC) and
      explicit congestion notification (ECN).
    - X722 supports only iWARP and a more limited set of configuration
      parameters.

Differences between adapters are described in each section of this document.

For both E810 and X722, the corresponding LAN driver (ice or i40e) must be
built from source included in this release and installed on your system prior
to installing irdma.

-------------
先决条件
-------------

- Compile and install the E810 or X722 LAN PF driver from source included in
  this release. Refer to the ice or i40e driver README for installation
  instructions.
    * For E810 adapters, use the ice driver.
    * For X722 adapters, use the i40e driver.
- For best results, use a fully supported OS from the Supported OS List below.
- For server memory requirements, see the "Memory Requirements" section of this
  document.
- Install required packages. Refer to the "Building" section of the rdma-core
  README for required packages for your OS:
        https://github.com/linux-rdma/rdma-core/blob/v27.0/README.md
    * RHEL 7 and SLES:
        Install all required packages listed in the rdma-core README.
    * RHEL 8:
        Install the required packages for RHEL 7, then install the following
        additional packages:
            dnf install python3-docutils perl-generators
    * Ubuntu:
        Install the required packages listed in the rdma-core README, then
        install the following additional package:
            apt-get install python3-docutils libsystemd-dev

* Note:
The following are sample repo files that can be used to get the dependent packages
for rdma-core. However, these may not be all that is required.

- For SLES
    http://download.opensuse.org/distribution/leap/42.3/repo/oss

- For RHEL 8.1
    http://vault.centos.org/8.1.1911/PowerTools/x86_64/os/

-----------------
支持的操作系统列表
-----------------

    Supported:
        * RHEL 8.3
        * RHEL 7.9
        * SLES 15 SP2
        * SLES 12 SP5
        * Ubuntu 18.04
        * Ubuntu 20.04

    Supported Not Validated:
        * RHEL 8.2
        * RHEL 8.1
        * RHEL 8
        * RHEL 7.8
        * RHEL 7.7
        * RHEL 7.6 + OFED 4.17-1
        * RHEL 7.5 + OFED 4.17-1
        * RHEL 7.4 + OFED 4.17-1
        * SLES 15 SP1
        * SLES 15 + OFED 4.17-1
        * SLES 12 SP 4 + OFED 4.17-1
        * SLES 12 SP 3 + OFED 4.17-1
        * Linux kernel stable 5.10.*
        * Linux kernel longterm 5.4.*, 4.19.*, 4.14.*

-------------------------

编译和安装

-------------------------

To build and install the irdma driver and supporting rdma-core libraries:

1.解压缩irdma驱动程序压缩包：
        tar zxf irdma-<version>.tgz

2.构建并安装RDMA驱动程序：
        cd irdma-<version>
        ./build.sh

   By default, the irdma driver is built using in-distro RDMA libraries and
   modules. Optionally, irdma may also be built using OFED modules. See the
   Supported OS List above for a list of OSes that support this option.
   * Note: Intel products are not validated on other vendors' proprietary
           software packages.
   To install irdma using OFED modules:
        - Download OFED-4.17-1.tgz from the OpenFabrics Alliance:
             wget http://openfabrics.org/downloads/OFED/ofed-4.17-1/OFED-4.17-1.tgz
        - Decompress the archive:
             tar xzvf OFED-4.17.1.tgz
        - Install OFED:
             cd OFED-4.17-1
             ./install --all
        - Reboot after installation is complete.
        - Build the irdma driver with the "ofed" option:
             cd /path/to/irdma-<version>
            ./build.sh ofed
        - Continue with the installation steps below.

3.加载驱动程序：
    RHEL and Ubuntu:
        modprobe irdma

    SLES:
        modprobe irdma --allow-unsupported

    Notes:
        - This modprobe step is required only during installation. Normally,
          irdma is autoloaded via a udev rule when ice or i40e is loaded:
             /usr/lib/udev/rules.d/90-rdma-hw-modules.rules
        - For SLES, to automatically allow loading unsupported modules, add the
          following to /etc/modprobe.d/10-unsupported-modules.conf:
              allow_unsupported_modules 1

4.卸载任何先前版本的rdma-core用户空间库。
   For example, in RHEL:
        yum erase rdma-core

    Note: "yum erase rdma-core" will also remove any packages that depend on
          rdma-core, such as perftest or fio. Please re-install them after
          installing rdma-core.

5.Patch，构建和安装rdma-core用户空间库：

    RHEL:   #1 从GitHub下载rdma-core-27.0.tar.gz
        wget https://github.com/linux-rdma/rdma-core/releases/download/v27.0/rdma-core-27.0.tar.gz      #2 Apply patch libirdma-27.0.patch to rdma-core
        tar -xzvf rdma-core-27.0.tar.gz
        cd rdma-core-27.0
        patch -p2 < /path/to/irdma-<version>/libirdma-27.0.patch  #别落了“<” 符号    #3 确保目录rdma-core / redhat和contents 位于“ root”组下
        cd ..
        chgrp -R root rdma-core-27.0/redhat #4 重新打包成适当的名称给building用 ( "tgz" 扩展名代替 "tar.gz")
        tar -zcvf rdma-core-27.0.tgz rdma-core-27.0     #5 构建 rdma-core
        mkdir -p ~/rpmbuild/SOURCES
        mkdir -p ~/rpmbuild/SPECS
        cp rdma-core-27.0.tgz ~/rpmbuild/SOURCES/
        cd ~/rpmbuild/SOURCES
        tar -xzvf rdma-core-27.0.tgz
        cp ~/rpmbuild/SOURCES/rdma-core-27.0/redhat/rdma-core.spec ~/rpmbuild/SPECS/
        cd ~/rpmbuild/SPECS/
        rpmbuild -ba rdma-core.spec #6 安装RPMs
        cd ~/rpmbuild/RPMS/x86_64
        yum install *27.0*.rpm

    SLES:
        # Download rdma-core-27.0.tar.gz from GitHub
        wget https://github.com/linux-rdma/rdma-core/releases/download/v27.0/rdma-core-27.0.tar.gz
        # Apply patch libirdma-27.0.patch to rdma-core
        tar -xzvf rdma-core-27.0.tar.gz
        cd rdma-core-27.0
        patch -p2 < /path/to/irdma-<version>/libirdma-27.0.patch
        cd ..
        # Zip the rdma-core directory into a tar.gz archive
        tar -zcvf rdma-core-27.0.tar.gz rdma-core-27.0
        # Create an empty placeholder baselibs.conf file
        touch /usr/src/packages/SOURCES/baselibs.conf
        # Build rdma-core
        cp rdma-core-27.0.tar.gz /usr/src/packages/SOURCES
        cp rdma-core-27.0/suse/rdma-core.spec /usr/src/packages/SPECS/
        cd /usr/src/packages/SPECS/
        rpmbuild -ba rdma-core.spec --without=curlmini
        cd /usr/src/packages/RPMS/x86_64
        rpm -ivh --force *27.0*.rpm

    Ubuntu:
        To create Debian packages from rdma-core:
        # Download rdma-core-27.0.tar.gz from GitHub
        wget https://github.com/linux-rdma/rdma-core/releases/download/v27.0/rdma-core-27.0.tar.gz
        # Apply patch libirdma-27.0.patch to rdma-core
        tar -xzvf rdma-core-27.0.tar.gz
        cd rdma-core-27.0
        patch -p2 < /path/to/irdma-<version>/libirdma-27.0.patch
        # B