DPDK编译及使用
DPDK版本
dpdk版本:DPDK 23.11.0(LTS)
下载链接:http://fast.dpdk.org/rel/dpdk-23.11.tar.xz
dpdk使用手册:http://doc.dpdk.org/guides/linux_gsg/
操作系统及软件包依赖
编译 + 运行需求
操作系统:
- 需求:N/A
- 当前:openEuler-22.03-LTS-SP2
- 查询命令:
cat /etc/os-release
- 注意:安装时候服务器用途选择server
内核版本:
- 要求:>= 4.14
- 当前:5.10.0-153.12.0.92
- 查询命令:
uname -a
GCC版本:
- 要求:>=5.0
- 当前:10.3.1
- 查询命令:
gcc --version
- 安装命令:
dnf install gcc
glibc
- 要求:>= 2.7
- 当前:2.34
- 查询命令:
ldd --version
- 安装命令:
dnf install glibc
numactl:
- 要求:N/A
- 当前:2.0.16-7
- 查询命令:
rpm -qa| grep numactl-devel
- 安装命令:
dnf install numactl numactl-libs numactl-devel
编译需求
pkg-config/pkgconf:
- 要求:>= 0.27
- 当前:1.8.0
- 查询命令:
pkg-config --version;pkgconf --version
- 安装命令:
dnf install pkg-config pkgconf
Python:
- 要求:>= 3.6
- 当前:3.9.9
- 查询命令:
python --version
- 安装命令:
dnf install python python3-devel
Meson:
- 要求:>= 0.53.2
- 当前:0.59.4
- 查询命令:
meson --version
- 安装命令:
dnf install meson
ninja:
- 要求: N/A
- 当前:1.10.2
- 查询命令:
ninja --version
- 安装命令:
dnf install ninja-build
pyelftools:
- 要求:>= 0.22
- 当前:0.27-1
- 查询命令:
rpm -qa| grep pyelftools
- 安装命令:
dnf install python-pyelftools
Mellanox固件升级和驱动升级
下载页面:https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/
需要的系统和版本:使用openeuler–> 22.03版本
编译安装
- 挂载镜像:
mkdir /mnt/mlx_ofed;mount -o loop ./MLNX_OFED_LINUX-23.10-0.5.5.0-openeuler22.03-x86_64.iso /mnt/mlx_ofed/;
- 驱动安装:
cd /mnt/mlx_ofed;
- 执行驱动安装:
./mlnxofedinstall --upstream-libs --dpdk --without-fw-update
;如果安装失败则执行下一步;如果安装成功则直接完成; - 安装编译依赖:
yum install createrepo
yum install elfutils-devel automake libtool rpm-build autoconf
yum install fuse-devel pciutils-devel
- 编译mlx5驱动并安装:
./mlnxofedinstall --add-kernel-support --without-fw-update (需要更新固件就去掉参数:--without-fw-update)
此时支持到mlx5 - 安装mlx4驱动(必须在第5步后边):
dnf install libibverbs libibverbs-devel
此时支持到mlx4 - 安装libpcap:
dnf install libpcap libpcap-devel
DPDK编译
- 解压dpdk并进入目录:
cd /home/;tar xvf dpdk-23.11.tar.xz -C /home/;cd /home/dpdk-23.11
- 配置build目录:
meson setup build
或者meson setup -Dexamples=l2fwd,l3fwd build
;在/home/dpdk-23.11目录下会生产build目录;
出现剩余如下模块跳过:
=================
Content Skipped
=================
apps:
libs:
drivers:
common/mvep: missing dependency, "libmusdk"
net/af_xdp: missing dependency, "libxdp >=1.2.2" and "libbpf"
net/ipn3ke: missing dependency, "libfdt"
net/mana: missing dependency, "mana"
net/mvneta: missing dependency, "libmusdk"
net/mvpp2: missing dependency, "libmusdk"
net/nfb: missing dependency, "libnfb"
net/sfc: missing dependency, "libatomic"
raw/ifpga: missing dependency, "libfdt"
crypto/armv8: missing dependency, "libAArch64crypto"
crypto/ipsec_mb: missing dependency, "libIPSec_MB"
crypto/mvsam: missing dependency, "libmusdk"
crypto/uadk: missing dependency, "libwd"
compress/isal: missing dependency, "libisal"
gpu/cuda: missing dependency, "cuda.h"
Build targets in project: 675
Found ninja-1.10.2 at /usr/bin/ninja
- 查看配置:
cd /home/dpdk-23.11/build;meson configure
- 配置运行平台:
meson configure -Dplatform=native
或者meson configure -Dplatform=soc_kunpeng920
- 配置最多CPU数:
meson configure -Dmax_lcores=256
- 配置安装路径:
meson configure -Dprefix=/home/dpdkinstall
- 编译:
cd /home/dpdk-23.11/build;ninja
- 安装:
ninja install
- 加载dpdk库:
export PKG_CONFIG_PATH=/home/dpdkinstall/lib64/pkgconfig/;ldconfig
- cflag获取:
pkg-config --cflags libdpdk
-I/home/dpdkinstall/include -include rte_config.h -march=native -mrtm -I/usr/include/libnl3
- libs共享库获取:
pkg-config --libs libdpdk
-Wl,--as-needed -L/home/dpdkinstall/lib64 -lrte_node -lrte_graph -lrte_pipeline -lrte_table -lrte_pdump -lrte_port -lrte_fib -lrte_pdcp -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_mldev -lrte_regexdev -lrte_rawdev -lrte_power -lrte_pcapng -lrte_member -lrte_lpm -lrte_latencystats -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_gpudev -lrte_dispatcher -lrte_eventdev -lrte_efd -lrte_dmadev -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bpf -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lrte_log
- libs静态库获取:
pkg-config --static --libs libdpdk
-Wl,--whole-archive -L/home/dpdkinstall/lib64 -l:librte_common_cpt.a -l:librte_common_dpaax.a -l:librte_common_iavf.a -l:librte_common_idpf.a -l:librte_common_octeontx.a -l:librte_bus_auxiliary.a -l:librte_bus_cdx.a -l:librte_bus_dpaa.a -l:librte_bus_fslmc.a -l:librte_bus_ifpga.a -l:librte_bus_pci.a -l:librte_bus_platform.a -l:librte_bus_vdev.a -l:librte_bus_vmbus.a -l:librte_common_cnxk.a -l:librte_common_mlx5.a -l:librte_common_nfp.a -l:librte_common_qat.a -l:librte_common_sfc_efx.a -l:librte_mempool_bucket.a -l:librte_mempool_cnxk.a -l:librte_mempool_dpaa.a -l:librte_mempool_dpaa2.a -l:librte_mempool_octeontx.a -l:librte_mempool_ring.a -l:librte_mempool_stack.a -l:librte_dma_cnxk.a -l:librte_dma_dpaa.a -l:librte_dma_dpaa2.a -l:librte_dma_hisilicon.a -l:librte_dma_idxd.a -l:librte_dma_ioat.a -l:librte_dma_skeleton.a -l:librte_net_af_packet.a -l:librte_net_ark.a -l:librte_net_atlantic.a -l:librte_net_avp.a -l:librte_net_axgbe.a -l:librte_net_bnx2x.a -l:librte_net_bnxt.a -l:librte_net_bond.a -l:librte_net_cnxk.a -l:librte_net_cpfl.a -l:librte_net_cxgbe.a -l:librte_net_dpaa.a -l:librte_net_dpaa2.a -l:librte_net_e1000.a -l:librte_net_ena.a -l:librte_net_enetc.a -l:librte_net_enetfec.a -l:librte_net_enic.a -l:librte_net_failsafe.a -l:librte_net_fm10k.a -l:librte_net_gve.a -l:librte_net_hinic.a -l:librte_net_hns3.a -l:librte_net_i40e.a -l:librte_net_iavf.a -l:librte_net_ice.a -l:librte_net_idpf.a -l:librte_net_igc.a -l:librte_net_ionic.a -l:librte_net_ixgbe.a -l:librte_net_memif.a -l:librte_net_mlx4.a -l:librte_net_mlx5.a -l:librte_net_netvsc.a -l:librte_net_nfp.a -l:librte_net_ngbe.a -l:librte_net_null.a -l:librte_net_octeontx.a -l:librte_net_octeon_ep.a -l:librte_net_pcap.a -l:librte_net_pfe.a -l:librte_net_qede.a -l:librte_net_ring.a -l:librte_net_softnic.a -l:librte_net_tap.a -l:librte_net_thunderx.a -l:librte_net_txgbe.a -l:librte_net_vdev_netvsc.a -l:librte_net_vhost.a -l:librte_net_virtio.a -l:librte_net_vmxnet3.a -l:librte_raw_cnxk_bphy.a -l:librte_raw_cnxk_gpio.a -l:librte_raw_dpaa2_cmdif.a -l:librte_raw_ntb.a -l:librte_raw_skeleton.a -l:librte_crypto_bcmfs.a -l:librte_crypto_caam_jr.a -l:librte_crypto_ccp.a -l:librte_crypto_cnxk.a -l:librte_crypto_dpaa_sec.a -l:librte_crypto_dpaa2_sec.a -l:librte_crypto_mlx5.a -l:librte_crypto_nitrox.a -l:librte_crypto_null.a -l:librte_crypto_octeontx.a -l:librte_crypto_openssl.a -l:librte_crypto_scheduler.a -l:librte_crypto_virtio.a -l:librte_compress_mlx5.a -l:librte_compress_octeontx.a -l:librte_compress_zlib.a -l:librte_regex_mlx5.a -l:librte_regex_cn9k.a -l:librte_ml_cnxk.a -l:librte_vdpa_ifc.a -l:librte_vdpa_mlx5.a -l:librte_vdpa_nfp.a -l:librte_vdpa_sfc.a -l:librte_event_cnxk.a -l:librte_event_dlb2.a -l:librte_event_dpaa.a -l:librte_event_dpaa2.a -l:librte_event_dsw.a -l:librte_event_opdl.a -l:librte_event_skeleton.a -l:librte_event_sw.a -l:librte_event_octeontx.a -l:librte_baseband_acc.a -l:librte_baseband_fpga_5gnr_fec.a -l:librte_baseband_fpga_lte_fec.a -l:librte_baseband_la12xx.a -l:librte_baseband_null.a -l:librte_baseband_turbo_sw.a -l:librte_node.a -l:librte_graph.a -l:librte_pipeline.a -l:librte_table.a -l:librte_pdump.a -l:librte_port.a -l:librte_fib.a -l:librte_pdcp.a -l:librte_ipsec.a -l:librte_vhost.a -l:librte_stack.a -l:librte_security.a -l:librte_sched.a -l:librte_reorder.a -l:librte_rib.a -l:librte_mldev.a -l:librte_regexdev.a -l:librte_rawdev.a -l:librte_power.a -l:librte_pcapng.a -l:librte_member.a -l:librte_lpm.a -l:librte_latencystats.a -l:librte_jobstats.a -l:librte_ip_frag.a -l:librte_gso.a -l:librte_gro.a -l:librte_gpudev.a -l:librte_dispatcher.a -l:librte_eventdev.a -l:librte_efd.a -l:librte_dmadev.a -l:librte_distributor.a -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_cfgfile.a -l:librte_bpf.a -l:librte_bitratestats.a -l:librte_bbdev.a -l:librte_acl.a -l:librte_timer.a -l:librte_hash.a -l:librte_metrics.a -l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a -l:librte_kvargs.a -l:librte_log.a -Wl,--no-whole-archive -Wl,--export-dynamic -lmtcr_ul -Wl,--as-needed -lrte_node -lrte_graph -lrte_pipeline -lrte_table -lrte_pdump -lrte_port -lrte_fib -lrte_pdcp -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_mldev -lrte_regexdev -lrte_rawdev -lrte_power -lrte_pcapng -lrte_member -lrte_lpm -lrte_latencystats -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_gpudev -lrte_dispatcher -lrte_eventdev -lrte_efd -lrte_dmadev -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bpf -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lrte_log -pthread -lm -ldl -lnuma -lmlx5 -libverbs -lbnxt_re-rdmav34 -lcxgb4-rdmav34 -lefa -lhns -lirdma-rdmav34 -lmlx4 -lmlx5 -lmthca-rdmav34 -locrdma-rdmav34 -lqedr-rdmav34 -lvmw_pvrdma-rdmav34 -lhfi1verbs-rdmav34 -lipathverbs-rdmav34 -lrxe-rdmav34 -lsiw-rdmav34 -lpthread -lnl-3 -lpthread -libverbs -lbnxt_re-rdmav34 -lcxgb4-rdmav34 -lefa -lhns -lirdma-rdmav34 -lmlx5 -lmthca-rdmav34 -locrdma-rdmav34 -lqedr-rdmav34 -lvmw_pvrdma-rdmav34 -lhfi1verbs-rdmav34 -lipathverbs-rdmav34 -lrxe-rdmav34 -lsiw-rdmav34 -libverbs -lpthread -lnl-3 -lpthread -lcrypto -ldl -pthread -lz -lmlx4 -libverbs -lbnxt_re-rdmav34 -lcxgb4-rdmav34 -lefa -lhns -lirdma-rdmav34 -lmlx4 -lmlx5 -lmthca-rdmav34 -locrdma-rdmav34 -lqedr-rdmav34 -lvmw_pvrdma-rdmav34 -lhfi1verbs-rdmav34 -lipathverbs-rdmav34 -lrxe-rdmav34 -lsiw-rdmav34 -libverbs -lpthread -lnl-route-3 -lnl-3 -lpthread -lpcap -lelf -lz
DPDK APP编译
测试main文件
/* SPDX-License-Identifier: BSD-3-Clause
* Copyright(c) 2010-2014 Intel Corporation
*/
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <errno.h>
#include <sys/queue.h>
//引入dpdk需要的头文件
#include <rte_memory.h>
#include <rte_launch.h>
#include <rte_eal.h>
#include <rte_per_lcore.h>
#include <rte_lcore.h>
#include <rte_debug.h>
/* Launch a function on lcore. 8< */
static int
lcore_hello(__rte_unused void *arg)
{
unsigned lcore_id;
lcore_id = rte_lcore_id();
printf("hello from core %u\n", lcore_id);
return 0;
}
/* >8 End of launching function on lcore. */
/* Initialization of Environment Abstraction Layer (EAL). 8< */
int
main(int argc, char **argv)
{
int ret;
unsigned lcore_id;
// 初始化函数
ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_panic("Cannot init EAL\n");
/* >8 End of initialization of Environment Abstraction Layer */
/* Launches the function on each lcore. 8< */
// 把lcore_hello函数注册给每一个cpu core
RTE_LCORE_FOREACH_WORKER(lcore_id) {
/* Simpler equivalent. 8< */
rte_eal_remote_launch(lcore_hello, NULL, lcore_id);
/* >8 End of simpler equivalent. */
}
/* call it on main lcore too */
// 主线程也调用一次lcore_hello
lcore_hello(NULL);
/* >8 End of launching the function on each lcore. */
// 等待其他core的线程退出
rte_eal_mp_wait_lcore();
// 清理dpdk
/* clean up the EAL */
rte_eal_cleanup();
return 0;
}
Makefile编写
先执行:export PKG_CONFIG_PATH=/home/dpdkinstall/lib64/pkgconfig/;ldconfig
APP = helloworld
SRCS-y := main.c
PKGCONF ?= pkg-config
#如果没有PKG_CONFIG_PATH则报错;需要设置dpdk安装路径到PKG_CONFIG_PATH,并执行ldconfig;
ifneq ($(shell $(PKGCONF) --exists libdpdk && echo 0), 0)
$(error "no installation of DPDK found")
endif
all: shared
.PHONY: shared static
shared: build/$(APP)-shared
ln -sf $(APP)-shared build/$(APP)
static: build/$(APP)-static
ln -sf $(APP)-static build/$(APP)
#获取libdpdk.pc的文件路径,放到APP的依赖中,如果此文件更新了,APP也会重新编译
PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
#获取头文件路径
CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
#获取共享库路径
LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
#获取静态库路径
LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
CFLAGS += -DALLOW_EXPERIMENTAL_API
#Makefile/PC_FILE发生变化了,APP也会重新编译
build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
build:
@mkdir -p $@
.PHONY: clean
clean:
rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
test -d build && rmdir -p build || true
编译命令: make
或者 make static
程序运行
- vmware虚拟机网卡添加
将添加的网卡驱动类型从e1000更改为vmxnet3;vmxnet3支持多队列,RSS,而且启动DPDK不会报错;
修改文件:Clone1-OpenEuler-22.03-LTS-SP2.vmx
ethernet0.virtualDev = "vmxnet3"
ethernet1.virtualDev = "vmxnet3"
ethernet2.virtualDev = "vmxnet3"
- 添加hugepage
/etc/default/grub
GRUB_CMDLINE_LINUX一行增加default_hugepagesz=1G hugepagesz=1G hugepages=4
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot
- 加载内核模块
modprobe vfio-pci #建议使用vfio,igb_uio有问题
- 获取网卡pciid
查看所有网卡:lspci | grep -i eth
查看网卡信息:lspci -nnks 0000:0b:00.0 -v
- 绑定网卡到vfio-pci
1. 接口down
ifconfig ens192 down
2. 如果不支持iommu可以进行关闭
echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
3. 更改dma_entry_limit,默认值为64k
echo 512000 > /sys/module/vfio_iommu_type1/parameters/dma_entry_limit
4. 如果支持iommu,需要在grub中配置, 第二步就不需要操作了
intel_iommu=on iommu=pt #intel架构开启iommu,使用pt
5. 接口绑定到vfio-pci
./dpdk-devbind.py --bind=vfio-pci ens192
或者
./dpdk-devbind.py -b vfio-pci 0000:0b:00.0
- vfio-pci支持sriov
1. 开启sriov
sudo modprobe vfio-pci enable_sriov=1 或者 echo 1 | sudo tee /sys/module/vfio_pci/parameters/enable_sriov
2. 网卡绑定到vfio-pci
./usertools/dpdk-devbind.py -b vfio-pci 0000:86:00.0
3. 创建2个vf
echo 2 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs
4. start pf
<build_dir>/app/dpdk-testpmd -l 22-25 -n 4 -a 86:00.0 \--vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=pf -- -i
5. start vf
<build_dir>/app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \--vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i
- 动态库路径依赖依赖:
export LD_LIBRARY_PATH=/home/dpdkinstall/lib64:$LD_LIBRARY_PATH
- 启动进程
./build/helloworld
[root@dpdk-dev home]# ./build/helloworld
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_vmxnet3 (15ad:07b0) device: 0000:0b:00.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
hello from core 1
hello from core 2
hello from core 3
hello from core 4
hello from core 5
hello from core 6
hello from core 7
hello from core 0
- 带参数的启动
GRUB中进行cpu隔离:
/etc/default/grub
isolcpus=1-3 nohz_full=1,2,3 irqaffinity=0,4,5,6,7
grub2-mkconfig -o /boot/grub2/grub.cfg
/home/build/helloworld -l1-3 -n 4 --main-lcore 3 -a 0b:00.0
-l:指定哪些CPU运行DPDK程序
--main-lcore:指定master进程运行的核,不指定运行在最小的coreid上
-s:设置service core,不会跑worker
-a:pci白名单,只要-a的pci设备会被dpdk接管