QEMU/KVM PCI Passthrough(82599ES) & DPDK 网络性能测试
IOMMU
& qemu-kvm
& hugepages
等具体配置在这里不再重复介绍, 可参考
QEMU/KVM PCI Passthrough(i350) & DPDK 网络性能测试.
这里只上传不同测试场景下性能数据.
原图链接
测试环境
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Model name: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Stepping: 7
CPU MHz: 1443.287
CPU max MHz: 3300.0000
CPU min MHz: 1200.0000
BogoMIPS: 5199.80
Virtualization: VT-x
L1d cache: 512 KiB
L1i cache: 512 KiB
L2 cache: 4 MiB
L3 cache: 40 MiB
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX vulnerable
Vulnerability Mds: Vulnerable; SMT vulnerable
Vulnerability Meltdown: Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon
pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt t
sc_deadline_timer aes xsave avx lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
kernel
启动参数:
[root@fc34 dpdk-wrks]# # grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-5.13.14-200.fc34.x86_64"
args="ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off $tuned_params isolcpus=1-15"
root="/dev/mapper/fedora_fedora-root"
initrd="/boot/initramfs-5.13.14-200.fc34.x86_64.img $tuned_initrd"
title="Fedora (5.13.14-200.fc34.x86_64) 34 (Server Edition)"
id="3e1208d6a2944886b8c800b19203e82f-5.13.14-200.fc34.x86_64"
NIC:
# lspci |grep Eth
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
NIC NUMA:
# cat /sys/bus/pci/devices/0000\:05\:00.0/numa_node
0
# cat /sys/bus/pci/devices/0000\:05\:00.1/numa_node
0
IXIA pktgen
宿主机(host) rxonly performance
#!/bin/sh
#ALLOW_DEVS="-a 02:00.3 "
ALLOW_DEVS+="-b 05:00.0 "
ALLOW_DEVS+="-a 05:00.1 "
TESTPMD_BIN=/root/devel/dpdk-21.05/v2011-snb/app/dpdk-testpmd
TESTPMD_BIN=dpdk-testpmd
TESTPMD_BIN=/devel/dpdk-21.08/v2108-sandybridge-gcc-11-50mlnx1-1.50218.x86_64/app/dpdk-testpmd
ARGS_EXTRA=" --huge-unlink "
ARGS_EXTRA+=" --iova-mode=pa "
numactl -N 0 -m 0 $TESTPMD_BIN -c 0xfe -m 4096 ${ALLOW_DEVS} -n 2 $ARGS_EXTRA \
-- -i --rxd=2048 --txd=2048 \
--rxq=3 --txq=3 --nb-cores=6 \
--forward-mode=rxonly \
--rss-ip --rss-udp \
--numa --rx-offloads=0 --mbcache=512 --burst=64 \
--mlockall -a
vfio with iommu=off intel_iommu=off
测试关闭IOMMU
& 开启 vfio enable_unsafe_noiommu_mode=1
RX 性能:
testpmd> show port stats 1
######################## NIC statistics for port 1 ########################
RX-packets: 266636840 RX-missed: 0 RX-bytes: 15998212086
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 14881458 Rx-bps: 7143102968
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>
vfio with iommu=on intel_iommu=on
测试开启IOMMU
& 开启 vfio
RX 性能:
testpmd> show port stats 1
######################## NIC statistics for port 1 ########################
RX-packets: 60085340 RX-missed: 151041955 RX-bytes: 3605124726
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 4087223 Rx-bps: 1961860888
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>
igb_uio
测试使用 igb_uio
RX 性能:
testpmd> show port stats 1
######################## NIC statistics for port 1 ########################
RX-packets: 70484884 RX-missed: 0 RX-bytes: 4229094726
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 14881470 Rx-bps: 7143108336
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>
宿主机(host) txonly performance
#!/bin/sh
#ALLOW_DEVS="-a 02:00.3"
#ALLOW_DEVS="-a 02:10.3"
ALLOW_DEVS+="-a 05:00.0 "
ALLOW_DEVS+="-a 05:00.1 "
TESTPMD_BIN=dpdk-testpmd
TESTPMD_BIN=/root/devel/dpdk-21.05/v2011-snb/app/dpdk-testpmd
TESTPMD_BIN=/devel/dpdk-21.08/v2108-sandybridge-gcc-11-50mlnx1-1.50218.x86_64/app/dpdk-testpmd
ARGS_EXTRA=" --iova-mode=pa --huge-unlink"
numactl -N 0 -m 0 $TESTPMD_BIN -c 0xfc -m 4096 ${ALLOW_DEVS} -n 2 $ARGS_EXTRA \
-- -i --rxd=2048 --txd=2048 \
--rxq=1 --txq=1 --nb-cores=4 \
--forward-mode=txonly --txpkts=60 \
--numa --txonly-multi-flow --tx-offloads=0 --mbcache=512 --burst=64 \
--mlockall -a
vfio-pci with iommu=off intel_iommu=off
测试关闭 IOMMU
& vfio enable_unsafe_noiommu_mode=1
TX 性能:
testpmd> show port stats 1
######################## NIC statistics for port 1 ########################
RX-packets: 1 RX-missed: 0 RX-bytes: 66
RX-errors: 0
RX-nombuf: 0
TX-packets: 165155071 TX-errors: 0 TX-bytes: 9909301124
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 14880763 Tx-bps: 7142767328
############################################################################
testpmd>
vfio-pci with iommu=on intel_iommu=on
测试开启 IOMMU
TX 性能:
testpmd> show port stats 1
######################## NIC statistics for port 1 ########################
RX-packets: 1 RX-missed: 0 RX-bytes: 66
RX-errors: 0
RX-nombuf: 0
TX-packets: 42082664 TX-errors: 0 TX-bytes: 2524958752
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 4163606 Tx-bps: 1998532944
############################################################################
testpmd>
igb_uio
测试使用 igb_uio
TX 性能:
testpmd>
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 1 RX-missed: 0 RX-bytes: 66
RX-errors: 0
RX-nombuf: 0
TX-packets: 218982941 TX-errors: 0 TX-bytes: 13138973388
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 14880768 Tx-bps: 7142768832
############################################################################
testpmd>
QEMU/KVM 虚拟机(guest) rxonly performance
iommu=off intel_iommu=off
测试关闭 IOMMU
& vfio enable_unsafe_noiommu_mode=1
RX 性能:
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 45891304 RX-missed: 110802769 RX-bytes: 2753482566
RX-errors: 0
RX-nombuf: 0
TX-packets: 1 TX-errors: 0 TX-bytes: 66
Throughput (since last show)
Rx-pps: 4305963 Rx-bps: 2066854672
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>
iommu=pt intel_iommu=on
测试开启 IOMMU
RX 性能:
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 23093640 RX-missed: 54535991 RX-bytes: 1385622966
RX-errors: 0
RX-nombuf: 0
TX-packets: 1 TX-errors: 0 TX-bytes: 66
Throughput (since last show)
Rx-pps: 4292522 Rx-bps: 2060408272
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>
QEMU/KVM 虚拟机(guset) txonly performance
iommu=off intel_iommu=off
测试关闭 IOMMU
& vfio enable_unsafe_noiommu_mode=1
TX 性能:
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 1 RX-missed: 0 RX-bytes: 66
RX-errors: 0
RX-nombuf: 0
TX-packets: 64208444 TX-errors: 0 TX-bytes: 3852506064
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 4264733 Tx-bps: 2047075928
############################################################################
testpmd>
iommu=pt intel_iommu=on
测试开启 IOMMU
TX 性能:
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 1 RX-missed: 0 RX-bytes: 66
RX-errors: 0
RX-nombuf: 0
TX-packets: 38054588 TX-errors: 0 TX-bytes: 2435493360
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 4238879 Tx-bps: 2170316088
############################################################################
testpmd> quit
结论
intel 82599ES
这款10G网卡在关闭iommu
模式下单核 RX&TX
在宿主机(host)均可达到小包线速.
开启iommu
模式下小包性能不是很理想, 具体原因待查.
后续1:
检查内核日志发现 DMAR
报错 DMAR: DRHD: handling fault status reg
, 估计是这个原因导致 82599ES
开启IOMMU
后性能降低, 具体原因待查.
# dmesg |grep "IOMMU\|DMAR"
[ 0.013139] ACPI: DMAR 0x000000007EF6A000 0000D8 (v01 INSYDE Romley 00000001 ACPI 00000001)
[ 0.013170] ACPI: Reserving DMAR table memory at [mem 0x7ef6a000-0x7ef6a0d7]
[ 0.257528] DMAR: IOMMU enabled
[ 0.469389] DMAR: Host address width 46
[ 0.469494] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[ 0.469607] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
[ 0.469742] DMAR: DRHD base: 0x000000ebffc000 flags: 0x1
[ 0.469853] DMAR: dmar1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
[ 0.469988] DMAR: RMRR base: 0x0000007ea8c000 end: 0x0000007ea94fff
[ 0.470101] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffe000 IOMMU 0
[ 0.470212] DMAR-IR: IOAPIC id 8 under DRHD base 0xebffc000 IOMMU 1
[ 0.470337] DMAR-IR: IOAPIC id 9 under DRHD base 0xebffc000 IOMMU 1
[ 0.470447] DMAR-IR: HPET id 0 under DRHD base 0xebffc000
[ 0.470556] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.471342] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.255591] DMAR: No ATSR found
[ 1.255700] DMAR: No SATC found
[ 1.255707] DMAR: dmar0: Using Queued invalidation
[ 1.256050] DMAR: dmar1: Using Queued invalidation
[ 1.338371] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 366.127902] DMAR: DRHD: handling fault status reg 2
[ 366.127914] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
[ 366.135408] DMAR: DRHD: handling fault status reg 102
[ 366.135422] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c7fb000 [fault reason 06] PTE Read access is not set
[ 396.401306] DMAR: DRHD: handling fault status reg 202
[ 396.401315] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
[ 396.408879] DMAR: DRHD: handling fault status reg 302
[ 396.408884] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
后续2:
重新加载默认BIOS设置后不再报DMAR
报错 DMAR: DRHD: handling fault status reg
错误, 但是开启IOMMU
后 82599ES
性能依然无法提升, 不知是否跟CPU有关系.
find /sys/kernel/iommu_groups/ -type l |grep 05\:
/sys/kernel/iommu_groups/25/devices/0000:05:00.1
/sys/kernel/iommu_groups/24/devices/0000:05:00.0
DPDK 19.11 Intel NIC Performance Report 在Intel® Xeon® Processor Platinum 8180 (38.5 M Cache, 2.50 GHz)
CPU下无此问题.