linux下vfio显卡透传

本文记录下在debian12环境下对显卡进行vfio透传。

背景

环境：debian主机，intel cpu带核显，额外有一NVIDIA GTX1060显卡
目的：将1060显卡透传给windows虚拟机使用

配置检查

内核config检查

内核需要支持vfio，

cat /boot/config-$(uname -r) | grep -i vfio
CONFIG_KVM_VFIO=y
CONFIG_VFIO=m
CONFIG_VFIO_GROUP=y
CONFIG_VFIO_CONTAINER=y
CONFIG_VFIO_IOMMU_TYPE1=m
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_VIRQFD=y
# CONFIG_VFIO_DEBUGFS is not set
# VFIO support for PCI devices
CONFIG_VFIO_PCI_CORE=m
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_IGD=y
# CONFIG_MLX5_VFIO_PCI is not set
# CONFIG_VIRTIO_VFIO_PCI is not set
# end of VFIO support for PCI devices
CONFIG_VFIO_MDEV=m

grub配置

修订/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

增加 iommu=pt或者intel_iommu=on
如果是amd的cpu，增加amd_iommu=on或者amd_iommu=pt

参考：https://docs.redhat.com/en/documentation/red_hat_virtualization/4.1/html/installation_guide/appe-configuring_a_hypervisor_host_for_pci_passthrough#appe-Configuring_a_Hypervisor_Host_for_PCI_Passthrough

If intel_iommu=on or amd_iommu=on works, you can try replacing them with iommu=pt or amd_iommu=pt. The pt option only enables IOMMU for devices used in passthrough and will provide better host performance. However, the option may not be supported on all hardware. Revert to previous option if the pt option doesn’t work for your host.
If the passthrough fails because the hardware does not support interrupt remapping, you can consider enabling the allow_unsafe_interrupts option if the virtual machines are trusted. The allow_unsafe_interrupts is not enabled by default because enabling it potentially exposes the host to MSI attacks from virtual machines. To enable the option:options vfio_iommu_type1 allow_unsafe_interrupts=1
设置完之后运行：sudo update-grub 下次开机生效。

modprobe配置

设置显卡驱动黑名单

查询显卡加载的驱动（这里已经加载了vfio）：

➜  modprobe.d lspci -nks 01:00.0     
01:00.0 0300: 10de:1c03 (rev a1)
        Subsystem: 1b4c:11d7
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
➜  modprobe.d lspci -nks 01:00.1
01:00.1 0403: 10de:10f1 (rev a1)
        Subsystem: 1b4c:11d7
        Kernel modules: snd_hda_intel

将驱动加入黑名单：

$ cat /etc/modprobe.d/nvidia-blacklist.conf 
blacklist nouveau
blacklist snd_hda_intel

设置vfio配置

查看显卡id：

$ lspci -nn | grep NVIDIA
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

加入配置：

$ cat /etc/modprobe.d/vfio.conf        
options vfio-pci ids=10de:1c03,10f1

配置完成之后执行：sudo update-initramfs -u 生效。

检查

重启，查看dmesg输出：

$ sudo dmesg | grep -i -e vfio
[   87.317168] VFIO - User Level meta-driver version: 0.3
[   87.326299] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[   87.326401] vfio_pci: add [10de:1c03[ffffffff:ffffffff]] class 0x000000/00000000
[   87.326403] vfio_pci: invalid id string "10f1"
[   88.469138] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

$ sudo dmesg | grep -e IOMMU   
[    0.071641] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.353552] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.353553] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.353554] DMAR: IOMMU feature nwfs inconsistent
[    0.353555] DMAR: IOMMU feature pasid inconsistent
[    0.353556] DMAR: IOMMU feature eafs inconsistent
[    0.353557] DMAR: IOMMU feature prs inconsistent
[    0.353557] DMAR: IOMMU feature nest inconsistent
[    0.353558] DMAR: IOMMU feature mts inconsistent
[    0.353558] DMAR: IOMMU feature sc_support inconsistent
[    0.353559] DMAR: IOMMU feature dev_iotlb_support inconsistent

运行

这里使用virt-manager来管理虚拟机，

注意，在透传时需要将同一个iommu group下面的设备都透传进去，查看哪些设备属于一个group可以参考下面的方法：

sudo dmesg | grep "iommu group"
[    0.353832] pci 0000:00:02.0: Adding to iommu group 0
[    0.353860] pci 0000:00:00.0: Adding to iommu group 1
[    0.353873] pci 0000:00:01.0: Adding to iommu group 2
[    0.353881] pci 0000:00:08.0: Adding to iommu group 3
[    0.353894] pci 0000:00:12.0: Adding to iommu group 4
[    0.353909] pci 0000:00:14.0: Adding to iommu group 5
[    0.353917] pci 0000:00:14.2: Adding to iommu group 5
[    0.353925] pci 0000:00:14.3: Adding to iommu group 6
[    0.353937] pci 0000:00:16.0: Adding to iommu group 7
[    0.353946] pci 0000:00:17.0: Adding to iommu group 8
[    0.353959] pci 0000:00:1b.0: Adding to iommu group 9
[    0.353968] pci 0000:00:1b.4: Adding to iommu group 10
[    0.353988] pci 0000:00:1c.0: Adding to iommu group 11
[    0.353998] pci 0000:00:1c.4: Adding to iommu group 12
[    0.354014] pci 0000:00:1d.0: Adding to iommu group 13
[    0.354023] pci 0000:00:1d.2: Adding to iommu group 14
[    0.354046] pci 0000:00:1f.0: Adding to iommu group 15
[    0.354055] pci 0000:00:1f.3: Adding to iommu group 15
[    0.354065] pci 0000:00:1f.4: Adding to iommu group 15
[    0.354074] pci 0000:00:1f.5: Adding to iommu group 15
[    0.354078] pci 0000:01:00.0: Adding to iommu group 2
[    0.354081] pci 0000:01:00.1: Adding to iommu group 2
[    0.354095] pci 0000:02:00.0: Adding to iommu group 16
[    0.354113] pci 0000:03:00.0: Adding to iommu group 17
[    0.354122] pci 0000:05:00.0: Adding to iommu group 18
[    0.354132] pci 0000:07:00.0: Adding to iommu group 19

参考

Ubuntu+KVM显卡透传
Appendix G. Configuring a Host for PCI Passthrough
How do I confirm that VFIO is working in 20.04?
KVM实战：原理、进阶与性能调优