测试结论
- kvm使用virtio虚拟网卡,网络小包处理能力损失了3/4。
测试过程
测试场景
- 本文接 KVM网络性能测试(1) 。
- 同样是16台服务器:同样是15台用作压测客户端,同样1台被测试。
- 同样是用工具 Ngrinder ,同样是测试 Tengine 的状态页面。
- 不同的是,这次我们用的品牌机、万兆网卡,并发用户数加大了。
- 用相同的测试方法:
- 测试物理机的网络性能(24C/32G)
tps = 227,115.4
mtt = 78.18
错误率 = 0.00% # 硬是不出错
虚拟用户 = 18000
- 分别测试1台虚拟机的网络性能(12C/8G、12C/8G)
tps = 49,650.8 、 49,724.9
mtt = 53.63 、 51.52
错误率 = 9.15% 、 9.19%
虚拟用 户= 18000 、18000
- 提高1台虚拟机的配置再测试(24C/8G、24C/8G)
tps = 37,318.6 、 37,380.5
mtt = 84.69 、 86.55
错误率 = 11.42% 、 11.33%
虚拟用户 = 18000 、18000
- 同时测试2台虚拟机的网络性能(12C/8G + 12C/8G)
tps = 53,280.6
mtt = 98.13
错误率 = 7.14%
虚拟用户 = 18000
此时宿主机资源并未完全耗尽:
其上两台虚拟机的资源也未耗尽,但st%
的占用表示有资源争抢:
测试结果
- 25万tps时:
pps/in = 1,237 k , pps/out = 742 k
。 - 宿主机 TPS超22万,虚拟机最高不到 6万。性能损失了3/4!
- 测试结果令人震惊,于是不敢相信地又跑了一次物理机测试,然而结果重现、还硬不报错。
测试环境
- 宿主机为戴尔M620,CPU=24核、内存=32G、硬盘=240G-ssd。
- 压测机器为15台同配置物理机,CPU=4核、内存=32G、硬盘=240G-ssd。
# libvirtd --version
libvirtd (libvirt) 1.2.9
# nginx -v
Tengine version: Tengine/2.1.1 (nginx/1.6.2)
# kvm --version
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u4), Copyright (c) 2003-2008 Fabrice Bellard
Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz * 24
Jessie-8.2 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64
Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
- 虚拟机的配置
virsh dumpxml kvm-net-perfmon
:
<domain type='kvm' id='7'>
<name>kvm-net-perfmon</name>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>12</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
</features>
<cpu mode='host-passthrough'>
<topology sockets='1' cores='12' threads='1'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='network' device='disk'>
<driver name='qemu' type='raw'/>
<auth username='higkoo'>
<secret type='ceph' uuid='666a519f-afae-0d6f-2233-99c88888ceb3'/>
</auth>
<source protocol='rbd' name='libvirt-pool/1018-kvm-net-perfmon'>
<host name='192.168.31.116' port='2233'/>
<host name='192.168.31.101' port='2233'/>
<host name='192.168.31.132' port='2233'/>
<host name='192.168.31.118' port='2233'/>
<host name='192.168.31.102' port='2233'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<iotune>
<total_iops_sec>800</total_iops_sec>
</iotune>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='bridge'>
<mac address='11:22:33:00:c6:a8'/>
<source bridge='br100'/>
<target dev='vnet1'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='2333' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</memballoon>
</devices>
</domain>