QEMU/KVM PCI Passthrough(82599ES) & DPDK 网络性能测试

QEMU/KVM PCI Passthrough(82599ES) & DPDK 网络性能测试

IOMMU & qemu-kvm & hugepages 等具体配置在这里不再重复介绍, 可参考
QEMU/KVM PCI Passthrough(i350) & DPDK 网络性能测试
.
这里只上传不同测试场景下性能数据.
在这里插入图片描述
原图链接

测试环境

CPU:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          32
On-line CPU(s) list:             0-31
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           45
Model name:                      Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Stepping:                        7
CPU MHz:                         1443.287
CPU max MHz:                     3300.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        5199.80
Virtualization:                  VT-x
L1d cache:                       512 KiB
L1i cache:                       512 KiB
L2 cache:                        4 MiB
L3 cache:                        40 MiB
NUMA node0 CPU(s):               0-7,16-23
NUMA node1 CPU(s):               8-15,24-31
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX vulnerable
Vulnerability Mds:               Vulnerable; SMT vulnerable
Vulnerability Meltdown:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon
                                  pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt t
                                 sc_deadline_timer aes xsave avx lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

kernel 启动参数:

[root@fc34 dpdk-wrks]# # grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-5.13.14-200.fc34.x86_64"
args="ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off $tuned_params isolcpus=1-15"
root="/dev/mapper/fedora_fedora-root"
initrd="/boot/initramfs-5.13.14-200.fc34.x86_64.img $tuned_initrd"
title="Fedora (5.13.14-200.fc34.x86_64) 34 (Server Edition)"
id="3e1208d6a2944886b8c800b19203e82f-5.13.14-200.fc34.x86_64"

NIC:

# lspci  |grep Eth
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

NIC NUMA:

# cat /sys/bus/pci/devices/0000\:05\:00.0/numa_node 
0
# cat /sys/bus/pci/devices/0000\:05\:00.1/numa_node 
0

IXIA pktgen

在这里插入图片描述

宿主机(host) rxonly performance

#!/bin/sh

#ALLOW_DEVS="-a 02:00.3 "
ALLOW_DEVS+="-b 05:00.0 "
ALLOW_DEVS+="-a 05:00.1 "

TESTPMD_BIN=/root/devel/dpdk-21.05/v2011-snb/app/dpdk-testpmd
TESTPMD_BIN=dpdk-testpmd
TESTPMD_BIN=/devel/dpdk-21.08/v2108-sandybridge-gcc-11-50mlnx1-1.50218.x86_64/app/dpdk-testpmd


ARGS_EXTRA=" --huge-unlink "
ARGS_EXTRA+=" --iova-mode=pa "

numactl -N 0 -m 0 $TESTPMD_BIN -c 0xfe -m 4096 ${ALLOW_DEVS} -n 2 $ARGS_EXTRA  \
	-- -i --rxd=2048 --txd=2048 \
	--rxq=3 --txq=3 --nb-cores=6 \
	--forward-mode=rxonly \
	--rss-ip --rss-udp \
	--numa  --rx-offloads=0 --mbcache=512 --burst=64 \
	--mlockall -a 

vfio with iommu=off intel_iommu=off

测试关闭IOMMU & 开启 vfio enable_unsafe_noiommu_mode=1 RX 性能:

testpmd> show port stats 1
  ######################## NIC statistics for port 1  ########################
  RX-packets: 266636840  RX-missed: 0          RX-bytes:  15998212086
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 0          TX-errors: 0          TX-bytes:  0

  Throughput (since last show)
  Rx-pps:     14881458          Rx-bps:   7143102968
  Tx-pps:            0          Tx-bps:            0
  ############################################################################
testpmd> 

vfio with iommu=on intel_iommu=on

测试开启IOMMU & 开启 vfio RX 性能:

testpmd> show port stats 1

  ######################## NIC statistics for port 1  ########################
  RX-packets: 60085340   RX-missed: 151041955  RX-bytes:  3605124726
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 0          TX-errors: 0          TX-bytes:  0

  Throughput (since last show)
  Rx-pps:      4087223          Rx-bps:   1961860888
  Tx-pps:            0          Tx-bps:            0
  ############################################################################
testpmd> 

igb_uio

测试使用 igb_uio RX 性能:

testpmd> show port stats 1
  ######################## NIC statistics for port 1  ########################
  RX-packets: 70484884   RX-missed: 0          RX-bytes:  4229094726
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 0          TX-errors: 0          TX-bytes:  0

  Throughput (since last show)
  Rx-pps:     14881470          Rx-bps:   7143108336
  Tx-pps:            0          Tx-bps:            0
  ############################################################################
testpmd> 

宿主机(host) txonly performance

#!/bin/sh

#ALLOW_DEVS="-a 02:00.3"
#ALLOW_DEVS="-a 02:10.3"
ALLOW_DEVS+="-a 05:00.0 "
ALLOW_DEVS+="-a 05:00.1 "

TESTPMD_BIN=dpdk-testpmd
TESTPMD_BIN=/root/devel/dpdk-21.05/v2011-snb/app/dpdk-testpmd
TESTPMD_BIN=/devel/dpdk-21.08/v2108-sandybridge-gcc-11-50mlnx1-1.50218.x86_64/app/dpdk-testpmd

ARGS_EXTRA=" --iova-mode=pa --huge-unlink"

numactl -N 0 -m 0 $TESTPMD_BIN -c 0xfc -m 4096 ${ALLOW_DEVS} -n 2 $ARGS_EXTRA  \
	-- -i --rxd=2048 --txd=2048 \
	--rxq=1 --txq=1 --nb-cores=4 \
	--forward-mode=txonly --txpkts=60 \
	--numa  --txonly-multi-flow --tx-offloads=0 --mbcache=512 --burst=64 \
	--mlockall -a 

vfio-pci with iommu=off intel_iommu=off

测试关闭 IOMMU & vfio enable_unsafe_noiommu_mode=1 TX 性能:

testpmd> show  port stats 1

  ######################## NIC statistics for port 1  ########################
  RX-packets: 1          RX-missed: 0          RX-bytes:  66
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 165155071  TX-errors: 0          TX-bytes:  9909301124

  Throughput (since last show)
  Rx-pps:            0          Rx-bps:            0
  Tx-pps:     14880763          Tx-bps:   7142767328
  ############################################################################
testpmd> 

vfio-pci with iommu=on intel_iommu=on

测试开启 IOMMU TX 性能:

testpmd> show port stats 1
  ######################## NIC statistics for port 1  ########################
  RX-packets: 1          RX-missed: 0          RX-bytes:  66
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 42082664   TX-errors: 0          TX-bytes:  2524958752

  Throughput (since last show)
  Rx-pps:            0          Rx-bps:            0
  Tx-pps:      4163606          Tx-bps:   1998532944
  ############################################################################
testpmd> 

igb_uio

测试使用 igb_uio TX 性能:

testpmd> 
testpmd> show  port stats  all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 1          RX-missed: 0          RX-bytes:  66
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 218982941  TX-errors: 0          TX-bytes:  13138973388

  Throughput (since last show)
  Rx-pps:            0          Rx-bps:            0
  Tx-pps:     14880768          Tx-bps:   7142768832
  ############################################################################
testpmd> 

QEMU/KVM 虚拟机(guest) rxonly performance

iommu=off intel_iommu=off

测试关闭 IOMMU & vfio enable_unsafe_noiommu_mode=1 RX 性能:

testpmd> show port stats all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 45891304   RX-missed: 110802769  RX-bytes:  2753482566
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 1          TX-errors: 0          TX-bytes:  66

  Throughput (since last show)
  Rx-pps:      4305963          Rx-bps:   2066854672
  Tx-pps:            0          Tx-bps:            0
  ############################################################################
testpmd> 

iommu=pt intel_iommu=on

测试开启 IOMMU RX 性能:

testpmd> show port stats  all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 23093640   RX-missed: 54535991   RX-bytes:  1385622966
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 1          TX-errors: 0          TX-bytes:  66

  Throughput (since last show)
  Rx-pps:      4292522          Rx-bps:   2060408272
  Tx-pps:            0          Tx-bps:            0
  ############################################################################
testpmd> 

QEMU/KVM 虚拟机(guset) txonly performance

iommu=off intel_iommu=off

测试关闭 IOMMU & vfio enable_unsafe_noiommu_mode=1 TX 性能:

testpmd> show port stats all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 1          RX-missed: 0          RX-bytes:  66
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 64208444   TX-errors: 0          TX-bytes:  3852506064

  Throughput (since last show)
  Rx-pps:            0          Rx-bps:            0
  Tx-pps:      4264733          Tx-bps:   2047075928
  ############################################################################
testpmd> 

iommu=pt intel_iommu=on

测试开启 IOMMU TX 性能:

testpmd> show port stats  all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 1          RX-missed: 0          RX-bytes:  66
  RX-errors: 0
  RX-nombuf:  0         
  TX-packets: 38054588   TX-errors: 0          TX-bytes:  2435493360

  Throughput (since last show)
  Rx-pps:            0          Rx-bps:            0
  Tx-pps:      4238879          Tx-bps:   2170316088
  ############################################################################
testpmd> quit

结论

intel 82599ES 这款10G网卡在关闭iommu模式下单核 RX&TX 在宿主机(host)均可达到小包线速.
开启iommu模式下小包性能不是很理想, 具体原因待查.

后续1:
检查内核日志发现 DMAR 报错 DMAR: DRHD: handling fault status reg , 估计是这个原因导致 82599ES 开启IOMMU后性能降低, 具体原因待查.

# dmesg  |grep "IOMMU\|DMAR"
[    0.013139] ACPI: DMAR 0x000000007EF6A000 0000D8 (v01 INSYDE Romley   00000001 ACPI 00000001)
[    0.013170] ACPI: Reserving DMAR table memory at [mem 0x7ef6a000-0x7ef6a0d7]
[    0.257528] DMAR: IOMMU enabled
[    0.469389] DMAR: Host address width 46
[    0.469494] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[    0.469607] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.469742] DMAR: DRHD base: 0x000000ebffc000 flags: 0x1
[    0.469853] DMAR: dmar1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.469988] DMAR: RMRR base: 0x0000007ea8c000 end: 0x0000007ea94fff
[    0.470101] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffe000 IOMMU 0
[    0.470212] DMAR-IR: IOAPIC id 8 under DRHD base  0xebffc000 IOMMU 1
[    0.470337] DMAR-IR: IOAPIC id 9 under DRHD base  0xebffc000 IOMMU 1
[    0.470447] DMAR-IR: HPET id 0 under DRHD base 0xebffc000
[    0.470556] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.471342] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.255591] DMAR: No ATSR found
[    1.255700] DMAR: No SATC found
[    1.255707] DMAR: dmar0: Using Queued invalidation
[    1.256050] DMAR: dmar1: Using Queued invalidation
[    1.338371] DMAR: Intel(R) Virtualization Technology for Directed I/O
[  366.127902] DMAR: DRHD: handling fault status reg 2
[  366.127914] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
[  366.135408] DMAR: DRHD: handling fault status reg 102
[  366.135422] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c7fb000 [fault reason 06] PTE Read access is not set
[  396.401306] DMAR: DRHD: handling fault status reg 202
[  396.401315] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
[  396.408879] DMAR: DRHD: handling fault status reg 302
[  396.408884] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set

后续2:
重新加载默认BIOS设置后不再报DMAR 报错 DMAR: DRHD: handling fault status reg错误, 但是开启IOMMU82599ES 性能依然无法提升, 不知是否跟CPU有关系.

find /sys/kernel/iommu_groups/ -type l |grep 05\:
/sys/kernel/iommu_groups/25/devices/0000:05:00.1
/sys/kernel/iommu_groups/24/devices/0000:05:00.0

DPDK 19.11 Intel NIC Performance ReportIntel® Xeon® Processor Platinum 8180 (38.5 M Cache, 2.50 GHz) CPU下无此问题.
在这里插入图片描述

  • 0
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值