KVM环境下vCPU绑定到物理CPU

KVM环境下vCPU绑定到物理CPU

KVM环境下vCPU绑定到物理CPU
在KVM环境中测试虚拟系统性能时，如果宿主机是有两个CPU socket的硬件，会碰到由于vCPU在不同物理CPU上浮动导致测试RFC2544时出现少量丢包的现象，测试结果非常不稳定。可以将vCPU绑定到物理CPU来优化这个问题(不能完全避免)，本章仅介绍单vCPU绑定到单物理CPU上的情况。绑定到单物理CPU上可以暂时规避少量丢包的问题，但是也存在宿主机上单个物理CPU满载后实际虚拟系统未到达性能上限的情况，本章也暂不深入。

1查看当前运行的虚拟系统名称
[root@localhost ~]# virsh list
setlocale: No such file or directory
Id Name State
----------------------------------------------------
4 vOS2 running

2、查看虚拟机的配置
[root@localhost ~]# virsh dumpxml vOS2
setlocale: No such file or directory
< domain type=‘kvm’ id=‘4’>
< name> vOS2< /name>
< uuid>9c31cd50-f719-4d35-90b2-3044dfdc95bb< /uuid>
< memory unit=‘KiB’>4194304< /memory>
< currentMemory unit=‘KiB’>4194304< /currentMemory>
< vcpu placement=‘static’>2< /vcpu>
< resource>
< partition>/machine< /partition>
< /resource>
< os>
< type arch=‘x86_64’ machine=‘pc-i440fx-rhel7.0.0’>hvm< /type>
< boot dev=‘hd’/>
< /os>
< features>
< acpi/>
< apic/>
< /features>
< cpu mode=‘host-model’>
< model fallback=‘allow’/>
< topology sockets=‘1’ cores=‘2’ threads=‘1’/>
< /cpu>
< clock offset=‘utc’>
< timer name=‘rtc’ tickpolicy=‘catchup’/>
< timer name=‘pit’ tickpolicy=‘delay’/>
< timer name=‘hpet’ present=‘no’/>
< /clock>
< on_poweroff>destroy< /on_poweroff>
< on_reboot>restart< /on_reboot>
< on_crash>restart< /on_crash>
< pm>
< suspend-to-mem enabled=‘no’/>
< suspend-to-disk enabled=‘no’/>
< /pm>
…省略大量信息

3、查看qemu进程号
[root@localhost ~]# ps -ef |grep vOS2
qemu 8480 1 51 16:43 ? 00:25:54 /usr/libexec/qemu-kvm -name vOS2 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu SandyBridge,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+osxsave,+f16c,+rdrand,+arat,+fsgsbase,+smep,+erms,+xsaveopt,+pdpe1gb -m 4096 -realtime mlock=off -smp 2,sockets=1,cores=2,threads=1 -uuid 9c31cd50-f719-4d35-90b2-3044dfdc95bb -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-vNGFW2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/vNGFW2.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:28:21:fe,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -global qxl-vga.vgamem_mb=16 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x8 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
root 10036 7775 0 17:34 pts/3 00:00:00 grep --color=auto vOS2

4、反复查看vcpu信息，最后一列是cpu的编号，可以看到qemu进程是在不同的CPU上不停的漂移的
[root@localhost ~]# ps -eLo ruser,pid,ppid,lwp,psr|grep 8480
qemu 8480 1 8480 3
qemu 8480 1 8491 2
qemu 8480 1 8494 4
qemu 8480 1 8505 9
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# ps -eLo ruser,pid,ppid,lwp,psr|grep 8480
qemu 8480 1 8480 6
qemu 8480 1 8491 2
qemu 8480 1 8494 4
qemu 8480 1 8505 9
[root@localhost ~]#
[root@localhost ~]# ps -eLo ruser,pid,ppid,lwp,psr|grep 8480
qemu 8480 1 8480 6
qemu 8480 1 8491 2
qemu 8480 1 8494 4
qemu 8480 1 8505 9
qemu 8480 1 10056 11

5、通过绑定进程到cpu比较麻烦，可以直接绑定vCPU到物理CPU。反复通过virsh命令查看vCPU对应的物理CPU，可以看到也是浮动的
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 2
State: running
CPU time: 107.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy //这里看到是宿主机所有物理CPU 核心，Y代表有可以使用，基于CPU时间片来回切换
VCPU: 1
CPU: 0
State: running
CPU time: 116.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy
[root@localhost ~]#
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 3
State: running
CPU time: 107.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy
VCPU: 1
CPU: 8
State: running
CPU time: 116.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 0
State: running
CPU time: 108.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy
VCPU: 1
CPU: 9
State: running
CPU time: 117.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy
6、绑定vCPU到物理CPU，这里将vCPU绑定到主CPU上
[root@localhost ~]# virsh vcpupin vOS2 0 2
setlocale: No such file or directory
[root@localhost ~]# virsh vcpupin vOS2 1 4
setlocale: No such file or directory
查看绑定结果，可以看到已经将vCPU0绑定到物理CPU2，vCPU1绑定到物理CPU4上
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 2
State: running
CPU time: 630.1s
CPU Affinity: --y-----------------
VCPU: 1
CPU: 4
State: running
CPU time: 880.7s
CPU Affinity: ----y---------------
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 2
State: running
CPU time: 693.0s
CPU Affinity: --y-----------------
VCPU: 1
CPU: 4
State: running
CPU time: 993.7s
CPU Affinity: ----y---------------
[root@localhost ~]# virsh dumpxml vOS2
setlocale: No such file or directory
< domain type=‘kvm’ id=‘4’>
< name>vOS2< /name>
< uuid>9c31cd50-f719-4d35-90b2-3044dfdc95bb< /uuid>
< memory unit=‘KiB’>4194304< /memory>
< currentMemory unit=‘KiB’>4194304< /currentMemory>
< vcpu placement=‘static’>2< /vcpu>
< cputune>
< vcpupin vcpu=‘0’ cpuset=‘2’/>
< vcpupin vcpu=‘1’ cpuset=‘4’/>
< /cputune>
< resource>
< partition>/machine< /partition>
< /resource>
< os>
< type arch=‘x86_64’ machine=‘pc-i440fx-rhel7.0.0’>hvm< /type>
< boot dev=‘hd’/>
< /os>
< features>
< acpi/>
< apic/>
< /features>
< cpu mode=‘host-model’>
< model fallback=‘allow’/>
< topology sockets=‘1’ cores=‘2’ threads=‘1’/>
< /cpu>
< clock offset=‘utc’>
< timer name=‘rtc’ tickpolicy=‘catchup’/>
< timer name=‘pit’ tickpolicy=‘delay’/>
< timer name=‘hpet’ present=‘no’/>
< /clock>
< on_poweroff>destroy< /on_poweroff>
< on_reboot>restart< /on_reboot>
< on_crash>restart< /on_crash>
< pm>

7、再次测试RFC2544性能，对比绑定前后结果：

未绑定CPU，连续测试64,512,1518字节时，测试到1518字节的时候会出现非常不稳定的丢包现象，需要单独测试1518字节才能得到一个测试结果

绑定CPU，多次测试，结果基本稳定