基于NXP iMX8M Mini处理器测试DPDK

By Toradex秦海

1). 简介

DPDK (Data Plane Development Kit) 软件是一组用户空间库和驱动程序，可加速在所有主要 CPU 架构上运行的网络数据包处理工作负载，以便提升整个网络数据服务的QoS。其最早由 Intel 大约 2010年创建，后由6WIND公司发展为开源社区版本后，极大的拓展了DPDK的影响力，现在DPDI已经是是 Linux 基金会旗下的开源项目之一，其支持涵盖了主流的CPU架构和繁多的NIC驱动以及虚拟化平台支持。

DPDK使用了轮询 (polling) 而不是中断来处理数据包。在收到数据包时，经 DPDK 重载的网卡驱动不会通过中断通知 CPU ，而是直接将数据包存入内存，交付应用层软件通过 DPDK 提供的接口来直接处理，这样节省了大量的 CPU 中断时间和内存拷贝时间。

本文所要测试的 NXP iMX8M Mini ARM SoC平台集成的 FEC NIC 在 DPDK 21.11 LTS发布版本提供了相关支持， 具体的 DPDK ENETFEC Poll Mode 驱动描述以及功能支持情况请见如下 dpdk.org 文档，本文就对其进行简单部署测试。

20. ENETFEC Poll Mode Driver — Data Plane Development Kit 23.03.0 documentation

本文所演示的平台来自于Toradex Verdin iMX8M Mini 嵌入式平台，基于 NXP iMX8M Mini 系列 ARM 处理器，核心为 Cortex-A53，包含一个支持 DPDK PMD 驱动的以太网接口。

2). 硬件准备

a). DUT-A : Verdin iMX8MM ARM 核心版配合 Dahlia Carrier Board 载板，并连接调试串口，后续测试中作为DPDK Packet Generator 设备。

b). DUT-B : Verdin iMX8MM ARM 核心版配合 Verdin Development Board 载板，并连接调试串口，后续测试中作为DPDK Packet Generator 设备。

c). Dahlia 载板 X11千兆网口和 Verdin Development Board X25 千兆网口通过一根网线直通连接。

 

 

3). 软件准备

a). Toradex Verdin iMXMM 模块标准 Ycoto Linux BSP 中没有包含 DPDK 相关库和驱动，需要通过 Ycoto Project/Openembedded 编译环境来进行添加编译，生成新的测试 Linux BSP Image。

b). 参考 Ycoto 编译环境来下载配置相关编译环境，使用最新的针对 Ycoto Linux BSP 6/Linux Kernel 5.15.x 版本的 branch

---------------------------------------

$ mkdir ${HOME}/oe-core

$ cd ${HOME}/oe-core

$ repo init -u git://git.toradex.com/toradex-manifest.git -b kirkstone-6.x.y -m tdxref/default.xml

$ repo sync

---------------------------------------

c). 由于默认的 oe-core/layers/meta-freescale/recipes-extended/dpdk/ 提供的 DPDK 19.11版本还未支持 iMX8MM，需要参考 NXP i.MX_Yocto_Project_User's_Guide 文档，下载 NXP imx-6.1.1-1.0.0 branch 代码来获取 DPDK 21.11 版本的 BB 描述文件。

---------------------------------------

$ mkdir ${HOME}/imx-yocto-bsp

$ cd ${HOME}/imx-yocto-bsp

$ repo init -u https://github.com/nxp-imx/imx-manifest -b imx-linux-langdale -m imx-6.1.1-1.0.0.xml

$ repo sync

---------------------------------------

d). 将 NXP imx-6.1.1-1.0.0 branch 包含的 DPDK 21.11 BB 文件复制到 Toradex Ycoto 环境

---------------------------------------

$ cd ${HOME}/oe-core

$ mkdir -p layers/meta-toradex-nxp/recipes-extended/dpdk

$ cp ${HOME}/imx-ycoto-bsp_6.1.1-1.0.0/sources/meta-imx/meta-sdk/recipes-extended/dpdk/dpdk* layers/meta-toradex-nxp/recipes-extended/dpdk/

$ $ ls layers/meta-toradex-nxp/recipes-extended/dpdk/

dpdk_21.11.bb  dpdk.inc

---------------------------------------

e). 修改配置文件后编译测试 Ycoto Linux BSP Image 文件

---------------------------------------

### Add below to ${HOME}/oe-core/build/conf/local.conf ###

# add dpdk package

IMAGE_INSTALL:append = " dpdk dpdk-examples dpdk-tools"

# accept freescale license

ACCEPT_FSL_EULA = "1"

### Compile BSP Image ###

$ MACHINE=verdin-imx8mm bitbake tdx-reference-multimedia-image

---------------------------------------

f). 参考这里的说明将编译生成的 Ycoto Linux Image 通过 Toradex Easy Installer 更新到两个测试平台的 Verdin iMX8MM 模块

g). 由于默认 Linux BSP 网口驱动还是 NXP 标准 Linux FEC Kernel驱动，需要通过修改 Linux Kernel Device Tree 来适配为 DPDK PMD 驱动。

./ 参考这里的说明下载 Linux Kernel 源代码， 对应 Branch 为 toradex_5.15-2.1.x-imx

./ 参考如下 Patch 文件修改 Device Tree 源代码，并重新编译生成所需的 Device Tree Binary 文件 imx8mm-verdin-wifi-dev.dtb

---------------------------------------

diff --git a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts

index ef952021832e..b2d60695ecf9 100644

--- a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts

+++ b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts

@@ -16,3 +16,7 @@

                     "toradex,verdin-imx8mm",

                     "fsl,imx8mm";

 };

+

+&fec1 {

+        compatible = "fsl,imx8mm-fec-uio";

+};

---------------------------------------

./ 将新的 imx8mm-verdin-wifi-dev.dtb 替换两个平台的 Verdin iMX8MM Linux /boot 下默认使用的 Device Tree 文件并重新启动，查看原本的 eth0 接口已经没有 probe 了

---------------------------------------

root@verdin-imx8mm-07276322:~# dmesg |grep eth

[    0.000000] psci: probing for conduit method from DT.

[    1.716513] fec-uio 30be0000.ethernet: UIO device full name imx-fec-uio initialized

[    8.382278] using random self ethernet address

[    8.387105] using random host ethernet address

root@verdin-imx8mm-07276322:~# ifconfig -a

can0: flags=128<NOARP>  mtu 16

        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 10  (UNSPEC)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        device interrupt 78  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 188  bytes 13742 (13.4 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 188  bytes 13742 (13.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

---------------------------------------

4). DPDK 示例应用测试

a). 首先配置 Huge Pages，可以通过设置下面 U-boot 环境变量来配置，这样就无需每次启动都配置一次

---------------------------------------

### configure huge paces ###

$ fw_setenv tdxargs ‘default_hugepagesz=2m hugepagesz=2m hugepages=448’

$ reboot

### check huge pages status ###

root@verdin-imx8mm-07276322:~# cat /proc/meminfo |grep -i '^hugepages'                          

HugePages_Total:     448

HugePages_Free:      448

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

---------------------------------------

b). 示例测试1 ： dpdk-testpmd 示例应用测试

./ dpdk-testpmd 应用配合的参数说明可以参考这里说明

./ DUT-B 通过 dpdk-testpmd 示例应用 Rx 模式作为 Packet Receiver

---------------------------------------

### DUT-B configure to Rx Mode ###

root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=rxonly                                                                   

EAL: Detected CPU lcores: 4

EAL: Detected NUMA nodes: 1

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: lsx_pciep:SoC(0xcc000000) not supported

EAL: Scan for (lsx_pciep) bus failed.

EAL: Selected IOVA mode 'PA'

EAL: No available 32768 kB hugepages reported

EAL: No available 64 kB hugepages reported

EAL: No available 1048576 kB hugepages reported

EAL: VFIO support initialized

TELEMETRY: No legacy callbacks, legacy socket not created

Interactive-mode selected

Set rxonly packet forwarding mode

Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p

arameters along with --numa.

testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)

Port 0: 01:01:01:01:01:01

Checking link statuses...

Done

### start forward mode ###

testpmd> start

rxonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m

ode: native

Logical Core 1 (socket 0) forwards packets on 1 streams:

  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  rxonly packet forwarding packets/burst=32

  nb forwarding cores=1 - nb forwarding ports=1

  port 0: RX queue number: 1 Tx queue number: 1

    Rx offloads=0x0 Tx offloads=0x0

    RX queue: 0

      RX desc=0 - RX free threshold=0

      RX threshold registers: pthresh=0 hthresh=0  wthresh=0

      RX Offloads=0x0

    TX queue: 0

      TX desc=0 - TX free threshold=0

      TX threshold registers: pthresh=0 hthresh=0  wthresh=0

      TX offloads=0x0 - TX RS bit threshold=0

---------------------------------------

./ DUT-A 通过 dpdk-testpmd 示例应用 Tx 模式作为 Packet Generator

---------------------------------------

### DUT-A configure to Tx Mode ###

root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=txonly

EAL: Detected CPU lcores: 4

EAL: Detected NUMA nodes: 1

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: lsx_pciep:SoC(0xcc000000) not supported

EAL: Scan for (lsx_pciep) bus failed.

EAL: Selected IOVA mode 'PA'

EAL: No available 32768 kB hugepages reported

EAL: No available 64 kB hugepages reported

EAL: No available 1048576 kB hugepages reported

EAL: VFIO support initialized

TELEMETRY: No legacy callbacks, legacy socket not created

Interactive-mode selected

Set txonly packet forwarding mode

Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p

arameters along with --numa.

testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)

Port 0: 01:01:01:01:01:01

Checking link statuses...

Done

### start forward mode ###

testpmd> start

txonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m

ode: native

Logical Core 1 (socket 0) forwards packets on 1 streams:

  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  txonly packet forwarding packets/burst=32

  packet len=64 - nb packet segments=1

  nb forwarding cores=1 - nb forwarding ports=1

  port 0: RX queue number: 1 Tx queue number: 1

    Rx offloads=0x0 Tx offloads=0x0

    RX queue: 0

      RX desc=0 - RX free threshold=0

      RX threshold registers: pthresh=0 hthresh=0  wthresh=0

      RX Offloads=0x0

    TX queue: 0

      TX desc=0 - TX free threshold=0

      TX threshold registers: pthresh=0 hthresh=0  wthresh=0

      TX offloads=0x0 - TX RS bit threshold=0

---------------------------------------

./ 测试一段时间后按顺序分别停止 DUT-A 和 DUT-B，查看统计结果，显示收发一致，工作正常

---------------------------------------

### Tx statistic on DUT-A ###

testpmd> stop

Telling cores to stop...

Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 23614688       TX-dropped: 0             TX-total: 23614688

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 23614688       TX-dropped: 0             TX-total: 23614688

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

### Rx statistic on DUT-B ###

testpmd> stop

Telling cores to stop...

Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 23614688       RX-dropped: 0             RX-total: 23614688

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 23614688       RX-dropped: 0             RX-total: 23614688

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

---------------------------------------

c). 示例测试2 ： dpdk-testpmd Tx Mode 配合 dpdk-l2fwd 示例应用测试

./ dpdk-l2fwd 应用配合的参数说明可以参考这里说明

./ DUT-B 通过 dpdk-l2fwd 配置为 L2 forward 模式

---------------------------------------

root@verdin-imx8mm-07275765:~# /usr/share/dpdk/examples/dpdk-l2fwd -c 0x1 -n 1 --vdev 'net_enetfec' -- -p 0x1 -P

---------------------------------------

// 注意使用 -P 参数来接受全部 MAC Address的包，否则不属于本机的会被自动丢弃

./ DUT-A 如示例 1 通过 dpdk-testpmd 示例应用 Tx 模式作为 Packet Generator 并 start 开始发包测试

---------------------------------------

root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --txd=1 --burst=1 --forward-mode=txonly

...

testpmd> start

...

---------------------------------------

// 注意，使用 --txd 和 --burst 来控制发包的速率，太快则 dpdk-l2fwd 无法及时处理

./ 测试一段时间后停止 DUT-A 发包，查看统计结果，显示收发一致，同时 DUT-B L2 IO 转发也一致，工作正常

---------------------------------------

### Tx statistic on DUT-A ###

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 13839386       TX-dropped: 41509766      TX-total: 55349152

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 13839386       TX-dropped: 83019532      TX-total: 96858918

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

### IO statistic on DUT-B ###

Port statistics ====================================

Statistics for port 0 ------------------------------

Packets sent:                 13839386

Packets received:             13839386

Packets dropped:                     0

Aggregate statistics ===============================

Total packets sent:           13839386

Total packets received:       13839386

Total packets dropped:               0

====================================================

---------------------------------------

d). 示例测试3 ： dpdk-testpmd IO Mode 示例应用测试

./ DUT-B 通过 dpdk-testpmd 配置为 IO forward 模式并 start 开始准备转发从 DUT-A 发送的包

---------------------------------------

root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io

...

testpmd> start

...

---------------------------------------

./ DUT-A 通过 dpdk-testpmd 配置为 IO forward 模式并通过 start tx_first开始进行先发一个包然后再 IO 收包的测试

---------------------------------------

root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io

...

testpmd> start tx_first

...

---------------------------------------

./ 测试一段时间后按顺序分别停止 DUT-A 和 DUT-B，查看统计结果，DUT-A 因为先发一组 Burst 包（ Burst 数值默认是32），因此 Tx 比 Rx 多32；同时 DUT-B IO 转发一致且和 DUT-A Tx 一致，工作正常

---------------------------------------

### IO statistic on DUT-A ###

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 10381092       RX-dropped: 0             RX-total: 10381092

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 10381092       RX-dropped: 0             RX-total: 10381092

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

### IO statistic on DUT-B ###

  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 10381124       RX-dropped: 0             RX-total: 10381124

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 10381124       RX-dropped: 0             RX-total: 10381124

  TX-packets: 10381124       TX-dropped: 0             TX-total: 10381124

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

---------------------------------------

5). 总结

本文基于NXP iMX8M Mini系列处理器简单演示了 DPDK 软件和驱动的部署以及示例应用测试，按照 NXP 官方说明， iMX8M Plus平台 ETH1 应该也支持 DPDK，只是目前在 toradex_5.15-2.0.x-imx branch 下驱动支持还存在一定问题，后续更高版本 branch 可能能够解决。

参考文档

NXP i.MX_Linux_Reference_Manual  Chapter 10 Data Plane Development Kit (DPDK)

20. ENETFEC Poll Mode Driver — Data Plane Development Kit 23.03.0 documentation