Xilinx zynqmp PL Ethernet Linux使用与优化

最新推荐文章于 2024-06-17 10:54:24 发布

三遍猪

最新推荐文章于 2024-06-17 10:54:24 发布

阅读量3.2k

点赞数 1

分类专栏： Xilinx Zynq/ZynqMP处理器开发文章标签： xilinx zynqmp ethernet linux

转载请关注我的-公众号-青儿创客基地-QQ技术交流群852283276

本文链接：https://blog.csdn.net/Zhu_Zhu_2009/article/details/90231333

版权

Xilinx Zynq/ZynqMP处理器开发专栏收录该内容

30 篇文章 40 订阅

订阅专栏

作者

QQ群：852283276
微信：arm80x86
微信公众号：青儿创客基地
B站：主页 https://space.bilibili.com/208826118

参考

Zynq-7000 AP SoC Performance – Gigabit Ethernet achieving the best performance
Linux AXI Ethernet driver
PS and PL based Ethernet in Zynq MPSoC
使用taskset将task绑定到指定的CPU上
 绑定CPU逻辑核心的利器——taskset
taskset
zynq 1G&10G 网络功能
 zynq-7000系列基于zynq-7015的vivado初步设计之linux下控制PL扩展的光以太网（1000BASE-X）
Linux下TC使用说明
 能查看TC filter的某个具体规则是否生效吗？
Linux网卡调优：RPS (Receive Packet Steering)
密集负载下的网卡中断负载均衡smp affinity及单队列RPS
Linux内核 RPS/RFS功能详细测试分析

使用

设备树，

dma {
	xlnx,include-dre;
};

ethernet {
	local-mac-address = [00 0a 35 66 44 c4];
	phy-mode = "xgmii";
};

优化

RPS/RFS

多队列网卡是一种技术，最初是用来解决网络IO QoS 问题的，随着网络IO的带宽的不断提升，单核CPU不能完全处满足网卡的需求，通过多队列网卡驱动的支持，将各个队列通过中断绑定到不同的核上。其实用bonding网卡绑定在一定程度也可以做中断负载均衡，两个网卡中断号可以绑定在不同的cpu核心上。由于RPS只是单纯把数据包均衡到不同的cpu，这个时候如果应用程序所在的cpu和软中断处理的cpu不是同一个，此时对于cpu cache的影响会很大，那么RFS确保应用程序处理的cpu跟软中断处理的cpu是同一个，这样就充分利用cpu的cache。示例脚本，

vi /opt/sbin/change_irq.sh
#!/bin/bash
ethtool -K eth0 gro off
ethtool -K eth1 gro off
ethtool -K eth2 gro off
ethtool -K eth3 gro off
service irqbalance stop
chkconfig irqbalance off
cat /proc/irq/{84,85,86,87,88,89,90,91,92,93}/smp_affinity
echo 1 > /proc/irq/84/smp_affinity
echo 2 > /proc/irq/85/smp_affinity
echo 4 > /proc/irq/86/smp_affinity
echo 8 > /proc/irq/87/smp_affinity
echo 10 > /proc/irq/88/smp_affinity
echo 20 > /proc/irq/89/smp_affinity
echo 40 > /proc/irq/90/smp_affinity
echo 80 > /proc/irq/91/smp_affinity
echo 100 > /proc/irq/92/smp_affinity
echo 200 > /proc/irq/93/smp_affinity
# Enable RPS (Receive Packet Steering)
rfc=4096
cc=$(grep -c processor /proc/cpuinfo)
rsfe=$(echo $cc*$rfc | bc)
sysctl -w net.core.rps_sock_flow_entries=$rsfe
for fileRps in $(ls /sys/class/net/eth*/queues/rx-*/rps_cpus)
do
echo fff > $fileRps
done
for fileRfc in $(ls /sys/class/net/eth*/queues/rx-*/rps_flow_cnt)
do
echo $rfc > $fileRfc
done
tail /sys/class/net/eth*/queues/rx-*/{rps_cpus,rps_flow_cnt}
chmod +x /opt/sbin/change_irq.sh
echo "/opt/sbin/change_irq.sh" >> /etc/rc.local

iperf3实测对性能无提升，反而下降（280MB/s->160MB/s），RPS/RFS解决的是很多连接/进程的业务处理场景，对单个TCP连接带宽无提升，

tc命令绑定队列

configQoS and/or fair queueing->Hardware Multiqueue-aware Multi Band Queuing。
The driver supports Linux multiqueue networking. It uses the alloc_etherdev_mq() function to allocate the subqueues for the device.

The userspace command ‘tc,’ part of the iproute2 package, is used to configure qdiscs. To add the MULTIQ qdisc assuming the device is called eth0, run the following command:

# tc qdisc add dev eth0 root handle 1: multiq

The qdisc will allocate the number of bands to equal the number of queues that the device reports, and bring the qdisc online.
Assuming eth0 has 4 Tx queues, the band mapping would look like:
band 0 => queue 0
band 1 => queue 1
band 2 => queue 2
band 3 => queue 3
The behavior of tc filters remains the same. However, a new tc action, skbedit, has been added.
Assuming we want to route all traffic to a specific host, for example 192.168.0.3, through a specific queue we could use this action and establish a filter such as:

 # tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dst 192.168.0.3 action skbedit queue_mapping 3

For details refer Linux kernel Documentation/networking/multiqueue.txt
实测对性能并无提高，

测试步骤

Xilinx XXV_MCDMA bencmarking steps，

绑核，加窗，

NOTE: Run iperf TX commands on ZynqMP and RX commands on the server machine.

a) TCP TX - MTU 1500
zynqmp# taskset -c 2 iperf3 -c SERVER_IP -T s1 -p 5101 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 3 iperf3 -c SERVER_IP -T s2 -p 5102 -t 60 -i 60 -b 2500M -Z &

b) UDP TX - MTU 1500
zynqmp# taskset -c 0 iperf3 -u -P 2 -c SERVER_IP -T s1 -p 5101 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 1 iperf3 -u -P 2 -c SERVER_IP -T s2 -p 5102 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 2 iperf3 -u -P 2 -c SERVER_IP -T s3 -p 5103 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 3 iperf3 -u -P 2 -c SERVER_IP -T s4 -p 5104 -t 60 -i 60 -b 2500M -Z &


c) TCP TX - MTU 9000
zynqmp# taskset -c 2 iperf3 -c SERVER_IP -T s1 -p 5101 -t 60 -i 60 -b 2500M &
zynqmp# taskset -c 3 iperf3 -c SERVER_IP -T s2 -p 5102 -t 60 -i 60 -b 2500M &


d) UDP TX - MTU 9000
zynqmp# taskset -c 0 iperf3 -P 2 -u -c SERVER_IP -T s1 -p 5101 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 1 iperf3 -P 2 -u -c SERVER_IP -T s2 -p 5102 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 2 iperf3 -P 2 -u -c SERVER_IP -T s3 -p 5103 -t 60 -i 60 -b 2500M -Z &
zynqmp# taskset -c 3 iperf3 -P 2 -u -c SERVER_IP -T s4 -p 5104 -t 60 -i 60 -b 2500M -Z &

e) TCP RX - MTU 9000
server# iperf3 -c SERVER_IP -T s1 -p 5101 -t 60 -i 60 -b 4000M -Z &
server# iperf3 -c SERVER_IP -T s2 -p 5102 -t 60 -i 60 -b 4000M -Z &

f) UDP RX - MTU 9000
server# iperf3 -c SERVER_IP  -u -T s1 -p 5101 -t 60 -i 60 -b 4000M -Z &
server# iperf3 -c SERVER_IP  -u -T s2 -p 5102 -t 60 -i 60 -b 4000M -Z &

g) UDP RX- MTU 1500
server# iperf3 -c SERVER_IP  -u -T s1 -p 5101 -t 60 -i 60 -b 4000M -Z &
server# iperf3 -c SERVER_IP  -u -T s2 -p 5102 -t 60 -i 60 -b 4000M -Z &

i) TCP RX- MTU 1500
RFS is disabled by default. To enable RFS, we must edit rps_sock_flow_entries and rps_flow_cnt.

zynqmp# echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
zynqmp# echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
zynqmp# echo 2048 >  /sys/class/net/eth0/queues/rx-1/rps_flow_cnt
For details refer : https://www.kernel.org/doc/Documentation/networking/scaling.txt

server# iperf3 -c SERVER_IP  -T s1 -p 5101 -t 60 -i 60 -b 4000M -Z &
server# iperf3 -c SERVER_IP  -T s2 -p 5102 -t 60 -i 60 -b 4000M -Z &

结果

加窗和应用绑核可以提高100MB/s左右，中断绑核实测反而会导致性能下降40MB/s，本身网卡的同一对发送和接收中断就应该在一个核上，避免切换，

root@zynqmp:~# iperf3 -c 192.168.8.16
Connecting to host 192.168.8.16, port 5201
[  4] local 192.168.8.192 port 39628 connected to 192.168.8.16 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   272 MBytes  2.28 Gbits/sec    0    392 KBytes       
[  4]   1.00-2.00   sec   273 MBytes  2.29 Gbits/sec    0    421 KBytes       
[  4]   2.00-3.00   sec   271 MBytes  2.27 Gbits/sec    0    433 KBytes       
[  4]   3.00-4.00   sec   272 MBytes  2.27 Gbits/sec    0    479 KBytes       
[  4]   4.00-5.00   sec   271 MBytes  2.28 Gbits/sec    0    479 KBytes       
[  4]   5.00-6.00   sec   272 MBytes  2.28 Gbits/sec    0    479 KBytes       
[  4]   6.00-7.00   sec   273 MBytes  2.29 Gbits/sec    0    479 KBytes       
[  4]   7.00-8.00   sec   272 MBytes  2.29 Gbits/sec    0    479 KBytes       
[  4]   8.00-9.00   sec   271 MBytes  2.27 Gbits/sec    0    479 KBytes       
[  4]   9.00-10.00  sec   271 MBytes  2.27 Gbits/sec    0    479 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.65 GBytes  2.28 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  2.65 GBytes  2.28 Gbits/sec                  receiver

iperf Done.
root@zynqmp:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.8.16, port 1040
[  5] local 192.168.8.192 port 5201 connected to 192.168.8.16 port 1041
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   217 MBytes  1.82 Gbits/sec                  
[  5]   1.00-2.00   sec   271 MBytes  2.27 Gbits/sec                  
[  5]   2.00-3.00   sec   271 MBytes  2.27 Gbits/sec                  
[  5]   3.00-4.00   sec   272 MBytes  2.28 Gbits/sec                  
[  5]   4.00-5.00   sec   273 MBytes  2.29 Gbits/sec                  
[  5]   5.00-6.00   sec   270 MBytes  2.27 Gbits/sec                  
[  5]   6.00-7.00   sec   271 MBytes  2.27 Gbits/sec                  
[  5]   7.00-8.00   sec   270 MBytes  2.27 Gbits/sec                  
[  5]   8.00-9.00   sec   271 MBytes  2.27 Gbits/sec                  
[  5]   9.00-10.00  sec   271 MBytes  2.27 Gbits/sec                  
[  5]  10.00-10.21  sec  56.9 MBytes  2.27 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.21  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.21  sec  2.65 GBytes  2.23 Gbits/sec                  receiver