搭建 Mellanox ConnectX-3 EN 10/40 Gbps 网卡 DPDK 运行环境

最新推荐文章于 2025-03-29 11:00:09 发布

功名半纸

最新推荐文章于 2025-03-29 11:00:09 发布

阅读量1.2w

点赞数 1

分类专栏：技术文摘

本文链接：https://blog.csdn.net/force_eagle/article/details/46728701

版权

技术文摘专栏收录该内容

62 篇文章

订阅专栏

DPD v2.0 当前依赖Mlx版本
Mellanox OFED 2.4-1.
Firmware version 2.33.5000 and higher.

1> 编译 mlx4 libraries
http://dpdk.org/download/mlx4/
因要使用 ibv_xxx 系列工具， ibv_xxx 不建议再使用DPDK.org下载的 mlx4 libraries. 可使用 MLNX OFED 内置 mlx4 libraries. 必须采用动态编译方式，方能运行正确.

编译完 MLNX OFED mxl4 libraries , 使用 make install DESTDIR=xxx , 并打包.

2> 编译DPDK v2.0.0

3> 编译mlx4 ofed 内核模块及mstflint程序
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
注意 MLNX_OFED 2.4-1.0.0 只支持以下OS与内核版本
- RHEL/CentOS 6.3      x86_64
- RHEL/CentOS 6.4      x86_64/PPC
- RHEL/CentOS 6.5      x86_64/PPC
- RHEL/CentOS 6.6      x86_64/PPC
- RHEL/CentOS 7.0      x86_64/PPC
- SLES11 SP2           x86_64/PPC
- SLES11 SP3           x86_64/PPC
- SLES12               x86_64
- OEL 6.4              x86_64
- OEL 6.5              x86_64
- OEL 6.6              x86_64
- Citrix XenServer Host 6.2    i686
- Fedora 19            x86_64
- Fedora 21            x86_64
- Ubuntu 12.04.4       x86_64
- Ubuntu 14.04         x86_64/PPC4le
- Ubuntu 14.10         x86_64/PPC4le
- Debian 6.0.9         x86_64
- Debian 7.5           x86_64
- Debian 7.6           x86_64
- kernel 3.10.48
- kernel 3.11.10
- kernel 3.14.12
- kernel 3.15.5
- kernel 3.16
- kernel 3.17
- kernel 3.18

Note: 建议使用RHEL相关的内核, 直接可以使用RPM安装相应模块.

编译mlx4内核模块
tar -zxvf mlnx-ofa_kernel-2.4.tgz
cd mlnx-ofa_kernel-2.4
# 配置，当前采用的是RHEL7第一版内核, 可用 yumdownloader --source kernel-3.10.0-123 下载.
./configure --with-mlx4_fc-mod  --with-mlx4_inf-mod --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mlx4-mod --with-mlx4_en-mod --with-ipoib-mod --with-srp-mod --with-rds-mod --with-iser-mod --kernel-version=3.10.0-mlx --modules-dir=/lib/modules/3.10.0-mlx/ --kernel-sources=/work/kernel-3.10.0-123.el7.src/linux-3.10.0-123.el7
make
make install # 建议安装，方便建立内核依赖关系

Note: 如为自定义内核配置，CONFIG_INFINIBAND 确认为 'n' 或者 'm', 否则此模块打包进内核在加载mlx4模块时会提示符号冲突.

编译 mft 内核模块，mstflint 使用.
tar -zxvf kernel-mft-3.8.0.tgz
cd kernel-mft-3.8.0
make KPVER=/work/kernel-3.10.0-123.el7.src/linux-3.10.0-123.el7
得到 mst_pciconf.ko  mst_pci.ko

编译 mstflint
tar -zxvf mstflint-3.8.0-1.27.gf3d39b6.tar.gz
cd mstflint-3.8.0
./configure --disable-inband --prefix=/usr
make install

4 . 加载模块

将模块打包并安装到目标机器下

modprobe ib_uverbs

modprobe mlx4_en

modprobe mlx4_core

modprobe mlx4_ib log_num_mgm_entry_size=-1

# 加载mst相关内核

insmod mst_pci.ko

insmod mst_pciconf.ko

运行 ibv_devinfo 获取设备信息

# ibv_devinfo -v

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.33.5100

node_guid: 0002:c903:00fa:a480

sys_image_guid: 0002:c903:00fa:a483

vendor_id: 0x02c9

vendor_part_id: 4099

hw_ver: 0x1

board_id: MT_1090110019

phys_port_cnt: 2

max_mr_size: 0xffffffffffffffff

page_size_cap: 0xfffffe00

max_qp: 392632

max_qp_wr: 16351

device_cap_flags: 0x005c9c76

BAD_PKEY_CNTR

BAD_QKEY_CNTR

AUTO_PATH_MIG

CHANGE_PHY_PORT

UD_AV_PORT_ENFORCE

PORT_ACTIVE_EVENT

SYS_IMAGE_GUID

RC_RNR_NAK_GEN

XRC

Unknown flags: 0x004c8000

device_cap_exp_flags: 0xD122001600000000

EXP_DEVICE_QPG

EXP_UD_RSS

EXP_MEM_WINDOW

EXP_MEM_MGT_EXTENSIONS

EXP_MW_TYPE_2B

EXP_CROSS_CHANNEL

EXP_MR_ALLOCATE

EXP_SHARED_MR

EXT_ATOMICS

Unknown flags: 0x8000000000000000

max_sge: 32

max_sge_rd: 0

max_cq: 65408

max_cqe: 4194303

max_mr: 524032

max_pd: 32764

max_qp_rd_atom: 16

max_ee_rd_atom: 0

max_res_rd_atom: 6282112

max_qp_init_rd_atom: 128

max_ee_init_rd_atom: 0

atomic_cap: ATOMIC_HCA (1)

log atomic arg sizes (mask) 8

max fetch and add bit boundary 64

log max atomic inline 3

max_ee: 0

max_rdd: 0

max_mw: 0

max_raw_ipv6_qp: 0

max_raw_ethy_qp: 0

max_mcast_grp: 131072

max_mcast_qp_attach: 244

max_total_mcast_qp_attach: 31981568

max_ah: 2147483647

max_fmr: 0

max_srq: 65472

max_srq_wr: 16383

max_srq_sge: 31

max_pkeys: 128

local_ca_ack_delay: 15

hca_core_clock: 427

max_klm_list_size: 0

max_send_wqe_inline_klms: 0

max_umr_recursion_depth: 0

max_umr_stride_dimension: 0

general_odp_caps:

rc_odp_caps:

NO SUPPORT

uc_odp_caps:

NO SUPPORT

ud_odp_caps:

NO SUPPORT

dc_odp_caps:

NO SUPPORT

xrc_odp_caps:

NO SUPPORT

raw_eth_odp_caps:

NO SUPPORT

max_dct: 0

port: 1

state: PORT_DOWN (1)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

max_msg_sz: 0x40000000

port_cap_flags: 0x00010000

max_vl_num: 2 (2)

bad_pkey_cntr: 0x0

qkey_viol_cntr: 0x0

sm_sl: 0

pkey_tbl_len: 1

gid_tbl_len: 128

subnet_timeout: 0

init_type_reply: 0

active_width: 4X (2)

active_speed: 10.0 Gbps (4)

phys_state: DISABLED (3)

GID[ 0]: fe80:0000:0000:0000:0202:c9ff:fefa:a481

port: 2

state: PORT_DOWN (1)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: InfiniBand

max_msg_sz: 0x40000000

port_cap_flags: 0x02504868

max_vl_num: 8 (4)

bad_pkey_cntr: 0x0

qkey_viol_cntr: 0x0

sm_sl: 0

pkey_tbl_len: 128

gid_tbl_len: 128

subnet_timeout: 0

init_type_reply: 0

active_width: 4X (2)

active_speed: 2.5 Gbps (1)

phys_state: POLLING (2)

GID[ 0]: fe80:0000:0000:0000:0002:c903:00fa:a482

ibv_devices 可获取设备相关 GUID

# ibv_devices

device node GUID

------ ----------------

mlx4_0 0002c90300faa480

connectx_port_config 在 mlnx-ofa_kernel-2.4-OFED.2.4.1.0.0.1 包的 ofed_scripts目录下，用于设置端口模式;

将端口配置成Ethernet模式;

# ./connectx_port_config

ConnectX PCI devices :

|----------------------------|

| 1 0000:02:00.0 |

|----------------------------|

Before port change:

auto (eth)

auto (ib)

|----------------------------|

| Possible port modes: |

| 1: Infiniband |

| 2: Ethernet |

| 3: AutoSense |

|----------------------------|

Select mode for port 1 (1,2,3): 2

Select mode for port 2 (1,2,3): 2

After port change:

eth

ifconfig 查看新增网卡设备.

# ifconfig -a

eth0 Link encap:Ethernet HWaddr 00:1E:67:9C:D4:90

inet addr:192.168.1.37 Bcast:192.168.1.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:872 errors:0 dropped:0 overruns:0 frame:0

TX packets:734 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:67141 (65.5 KiB) TX bytes:82711 (80.7 KiB)

Memory:d0b20000-d0b40000

eth1 Link encap:Ethernet HWaddr 00:1E:67:9C:D4:91

BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Memory:d0b00000-d0b20000

eth2 Link encap:Ethernet HWaddr 00:02:C9:FA:A4:81

BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth3 Link encap:Ethernet HWaddr 00:02:C9:FA:A4:82

BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:225 errors:0 dropped:0 overruns:0 frame:0

TX packets:225 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:25200 (24.6 KiB) TX bytes:25200 (24.6 KiB)

查看新增网卡驱动属性.

ethtool -i eth2

driver: mlx4_en

version: 2.4-1.0.0 (Apr 22 2015)

firmware-version: 2.33.5100

bus-info: 0000:02:00.0

ethtool -i eth3

driver: mlx4_en

version: 2.4-1.0.0 (Apr 22 2015)

firmware-version: 2.33.5100

bus-info: 0000:02:00.0

由此mlx4加载成功.

5> 运行测试程序

./testpmd -c 0xffff -m 4096 -n 1 -r 1 -- --rx=1 --tx=2 -i

set fwd txonly

start all

show port stats all

./trafgen --port0 port0.conf

参考:

http://dpdk.org/doc/guides/nics/mlx4.html