Ryzen 4800H KVM Bad Cache Performance

Edit 20200717: The cache performance is fixed. But PUBG performance remains bad!

Here is my updated config:

 

sudo chrt -r 1 taskset -c 4-15 qemu-system-x86_64 \
  -drive if=pflash,format=raw,readonly,file=$VGAPT_FIRMWARE_BIN \
  -drive if=pflash,format=raw,file=$VGAPT_FIRMWARE_VARS_TMP \
  -enable-kvm \
  -machine q35,accel=kvm,mem-merge=off \
  -cpu host,kvm=off,topoext=on,host-cache-info=on,hv_relaxed,hv_vapic,hv_time,hv_vpindex,hv_synic,hv_frequencies,hv_vendor_id=1234567890ab,hv_spinlocks=0x1fff \
  -smp 12,sockets=1,cores=6,threads=2 \
  -m 12288 \
  -mem-prealloc \
  -mem-path /dev/hugepages \
  -vga none \
  -rtc base=localtime \
  -boot menu=on \
  -acpitable file=/home/blabla/kvm/SSDT1.dat \
  -device vfio-pci,host=01:00.0 \
  -device vfio-pci,host=01:00.1 \
  -device vfio-pci,host=01:00.2 \
  -device vfio-pci,host=01:00.3 \
  -drive file=/dev/nvme0n1p3,format=raw,if=virtio,cache=none,index=0 \
  -drive file=/dev/nvme1n1p4,format=raw,if=virtio,cache=none,index=1 \
  -usb -device usb-host,hostbus=3,hostaddr=2 \
  -usb -device usb-host,hostbus=3,hostaddr=3 \
  -usb -device usb-host,hostbus=5,hostaddr=3 \
;

I also tried libvirt CPU pinning and I see no improvement.

lscpu -e output:

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ
  0    0      0    0 0:0:0:0          yes 2900.0000 1400.0000
  1    0      0    0 0:0:0:0          yes 2900.0000 1400.0000
  2    0      0    1 1:1:1:0          yes 2900.0000 1400.0000
  3    0      0    1 1:1:1:0          yes 2900.0000 1400.0000
  4    0      0    2 2:2:2:0          yes 2900.0000 1400.0000
  5    0      0    2 2:2:2:0          yes 2900.0000 1400.0000
  6    0      0    3 3:3:3:0          yes 2900.0000 1400.0000
  7    0      0    3 3:3:3:0          yes 2900.0000 1400.0000
  8    0      0    4 4:4:4:1          yes 2900.0000 1400.0000
  9    0      0    4 4:4:4:1          yes 2900.0000 1400.0000
 10    0      0    5 5:5:5:1          yes 2900.0000 1400.0000
 11    0      0    5 5:5:5:1          yes 2900.0000 1400.0000
 12    0      0    6 6:6:6:1          yes 2900.0000 1400.0000
 13    0      0    6 6:6:6:1          yes 2900.0000 1400.0000
 14    0      0    7 7:7:7:1          yes 2900.0000 1400.0000
 15    0      0    7 7:7:7:1          yes 2900.0000 1400.0000

My libvirt CPU pinning config.

  <cputune>                                                                              
    <vcpupin vcpu='0' cpuset='4'/>                                                       
    <vcpupin vcpu='1' cpuset='5'/>                                                       
    <vcpupin vcpu='2' cpuset='6'/>                                                       
    <vcpupin vcpu='3' cpuset='7'/>                                                       
    <vcpupin vcpu='4' cpuset='8'/>                                                       
    <vcpupin vcpu='5' cpuset='9'/>                                                       
    <vcpupin vcpu='6' cpuset='10'/>                                                      
    <vcpupin vcpu='7' cpuset='11'/>                                                      
    <vcpupin vcpu='8' cpuset='12'/>                                                      
    <vcpupin vcpu='9' cpuset='13'/>                                                      
    <vcpupin vcpu='10' cpuset='14'/>                                                     
    <vcpupin vcpu='11' cpuset='15'/>                                                     
    <emulatorpin cpuset='0-1'/>                                                          
    <iothreadpin iothread='1' cpuset='2-3'/>                                             
  </cputune>

I have also tried turning all the melt-down mitigations off for kernel boot parameters-> no obviously improvement.

#######################################################################

# ORIGINAL POST #

Hi!

So I have an ASUS TUF A15 Laptop with AMD 4800H and RTX 2060.

I did the usual KVM GPU passthrough and some performance tuning. The GPU passthrough is perfect but the the CPU, especial the cache IO and latency is really bad. This does not affect normal use but in CPU-heavy FPS games this is like a nightmare with less than half of the performance. Asking for HELP! I used to have an Intel machine and the VM performance is basically lossless with same games.

Benchmark comparison:

 

Bare Metal Windows 2004

 

 

VM Windows 2004

 

What I did for performance tuning:

1. CPU pinning (using taskset): tried (1) last 4 cores 8 threads (2) last 6 cores 12 threads (3) 3 cores per CCX = 6 cores 12 threads.

2. Hugepages (of course)

3. some hypervisor enlightments (see below for the flags)

4. set CPU performance governor to "performance" (This increased in game FPS by 15%, however still very bad)

5. I also tried setting cpu model to be "EPYC", no discernible difference.

My qemu config:

taskset 0xFFF0 qemu-system-x86_64 \
-drive if=pflash,format=raw,readonly,file=$VGAPT_FIRMWARE_BIN \
-drive if=pflash,format=raw,file=$VGAPT_FIRMWARE_VARS_TMP \
-enable-kvm \
-machine q35,accel=kvm,mem-merge=off \
-cpu host,kvm=off,topoext=on,hv_relaxed,hv_vapic,hv_time,hv_vpindex,hv_synic,hv_vendor_id=1234567890ab,hv_spinlocks=0x1fff \
-smp 12,sockets=1,cores=6,threads=2 \
-m 16384 \
-mem-prealloc \
-mem-path /dev/hugepages \
-vga none \
-rtc base=localtime \
-boot menu=on \
-acpitable file=/home/blabla/kvm/SSDT1.dat \
-device vfio-pci,host=01:00.0,romfile=/home/blabla/kvm/TU106.rom \
-device vfio-pci,host=01:00.1 \
-device vfio-pci,host=01:00.2 \
-device vfio-pci,host=01:00.3 \
-drive file=/dev/nvme0n1p7,format=raw,if=virtio,cache=none,index=0 \
-drive file=/dev/nvme1n1p4,format=raw,if=virtio,cache=none,index=1 \
-usb -device usb-host,hostbus=3,hostaddr=2 \
-usb -device usb-host,hostbus=3,hostaddr=3 \
-usb -device usb-host,hostbus=5,hostaddr=3 \
;

Thanks in advance!

Edit: Add lstopo output.

 

 

4800h lstopo output

Edit 2: FIXED cache latency issue! Gaming performance improves a bit. Still very bad performance in PUBG. Needs further investigation.

Fix: Make sure you use qemu version newer than qemu 4.1. Then add `host-cache-info=on` to `qemu -cpu` command.

 

https://www.reddit.com/r/VFIO/comments/hga3ho/ryzen_4800h_kvm_bad_cache_performance/