proxmox 直通显卡（GPU）

零 修订记录

序号	修订内容	修订时间
1	新增	20210622

一 摘要

本文主要介绍基于proxmox 虚拟机 直通显卡，最终希望在该宿主机上建两个虚机，各自直通一张gpu 卡，两台虚机同时使用。

二 环境信息

（一）物理机信息

品牌	型号	出厂配置	修改配置
浪潮	SA5212M5	42102/128G/SSD:240G2/SAS:2T 7.2K 2 /10G X7102/1G PHY卡*1/RAID卡 SAS3108 2GB	去掉一张万兆光卡，插上两张英伟达 T4

GPU 卡信息

root@testgpu:/etc# lspci -nnk | grep "NVIDIA"
3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
root@testgpu:/etc#

（二） 软件信息

proxmox 6.4.4

三 实施

Intel 机器和AMD 机器 配置文件等相关命令是有区别的。

(一) Intel 机器 gpu passthrough

3.1.1 检查宿主机gpu 卡信息

经检查 该机器有两张nvidia T4

root@testgpu:/etc# lspci -nnk | grep "NVIDIA"
3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
root@testgpu:/etc#

3.1.2 Enable the IOMMU

3.1.2.1 编辑/etc/default/grub

先备份

root@testgpu:/etc# cp /etc/default/grub /etc/default/grub.bak.orig
root@testgpu:/etc#

修改

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream video=vesafb:off video=efifb:off"

修改说明：
pcie_acs_override=downstream 用于拆分iommu，pcie 一二槽同时插上时会共享带宽，如果不拆分iommu，当两个设备同时直通，其中一个就会显示占用

video=vesafb:off video=efifb:off ：禁止启动和vesa驱动和efi启动的显卡

3.1.2.2 update-grub && reboot

root@testgpu:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.106-1-pve
Found initrd image: /boot/initrd.img-5.4.106-1-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
root@testgpu:~#

root@testgpu:~# reboot

3.1.2.3 Verify IOMMU is enabled

root@testgpu:~# dmesg | grep -e DMAR -e IOMMU

There should be a line that looks like “DMAR: IOMMU enabled”. If there is no output, something is wrong.

root@testgpu:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.009867] ACPI: DMAR 0x000000006AA82C08 0002A0 (v01 ALASKA A M I    00000001 INTL 20091013)
[    0.527674] DMAR: IOMMU enabled //找到这句
[    0.902757] DMAR: Host address width 46
[    0.902758] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
[    0.902765] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902766] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[    0.902770] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902771] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0
[    0.902776] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902777] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.902780] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902781] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0
[    0.902784] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902785] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0
[    0.902788] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902789] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0
[    0.902792] DMAR: dmar6: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902792] DMAR: DRHD base: 0x0000009d7fc000 flags: 0x1
[    0.902795] DMAR: dmar7: reg_base_addr 9d7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.902796] DMAR: RMRR base: 0x0000006d1ea000 end: 0x0000006d1fafff
[    0.902797] DMAR: ATSR flags: 0x0
[    0.902798] DMAR: ATSR flags: 0x0

如果该命令没有输出，可能是VT-d 没有开启，需要到bios 里设置下。

3.1.3 Required Modules

修订 /etc/modules
先备份

root@testgpu:~# cp /etc/modules /etc/modules.bak.orig

新增如下内容

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

3.1.4 IOMMU Interrupt Remapping

root@testgpu:~# dmesg | grep 'remapping'
[    0.902812] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.904778] DMAR-IR: Enabled IRQ remapping in x2apic mode  //关键句
root@testgpu:~#

如果系统不支持interrupt remapping ，请配置

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

我这里是支持的。

3.1.5 Verify IOMMU Isolation

For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.

You should have something like:

root@testgpu:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/108/devices/0000:85:1e.5
/sys/kernel/iommu_groups/108/devices/0000:85:1e.3
/sys/kernel/iommu_groups/108/devices/0000:85:1e.1
/sys/kernel/iommu_groups/108/devices/0000:85:1e.6
/sys/kernel/iommu_groups/108/devices/0000:85:1e.4
/sys/kernel/iommu_groups/108/devices/0000:85:1e.2
/sys/kernel/iommu_groups/108/devices/0000:85:1e.0
/sys/kernel/iommu_groups/55/devices/0000:3a:0b.2

3.1.6 GPU Passthrough

3.1.6.1 HOST Configuration

宿主机需要禁用这张gpu卡 ，有两个方法

3.1.6.1.1 pass the device id 通vfion-pci

root@testgpu:~# lspci | grep NVIDIA
3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
af:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
root@testgpu:~# lspci -n -s 3b:00
3b:00.0 0302: 10de:1eb8 (rev a1)
root@testgpu:~# lspci -n -s af:00
af:00.0 0302: 10de:1eb8 (rev a1)
root@testgpu:~#

gpu 卡型号写到 /etc/modprobe.d/vfio.conf

root@testgpu:/etc/modprobe.d# ls
pve-blacklist.conf
root@testgpu:/etc/modprobe.d# less pve-blacklist.conf

[1]+  Stopped                 less pve-blacklist.conf
root@testgpu:/etc/modprobe.d# echo "options vfio-pci ids=10de:1eb8 " > /etc/modprobe.d/vfio.conf
root@testgpu:/etc/modprobe.d# pwd
/etc/modprobe.d
root@testgpu:/etc/modprobe.d#

3.1.6.1.2 禁用驱动

与3.1.6.1.1 二选一即可，不过我这次操作 是两步都做了
修改/etc/modprobe.d/pve-blacklist.conf

先备份

root@testgpu:/etc/modprobe.d# cp /etc/modprobe.d/pve-blacklist.conf /etc/modprobe.d/pve-blacklist.conf.bak.orig
root@testgpu:/etc/modprobe.d#

root@testgpu:/etc/modprobe.d# echo "blacklist radeon" >> /etc/modprobe.d/pve-blacklist.conf
root@testgpu:/etc/modprobe.d# echo "blacklist nouveau" >> /etc/modprobe.d/pve-blacklist.conf
root@testgpu:/etc/modprobe.d# echo "blacklist nvidia" >> /etc/modprobe.d/pve-blacklist.conf

3.1.6.1.3 更新配置 重启机器

root@testgpu:/etc/modprobe.d# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.4.106-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. // 这个错误直接忽略，因为proxmox 安装的debian 并不是一个完整的系统引起
root@testgpu:/etc/modprobe.d#

root@testgpu:/etc/modprobe.d# reboot

root@testgpu:~# lspci -nnk | grep NVI
3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
root@testgpu:~#

3.1.6.2 VM Configuration

将设备传给vm
需要在设备宿主机上执行

qm set 100 -hostpci0 3b:00.0