零 修订记录
序号 | 修订内容 | 修订时间 |
---|---|---|
1 | 新增 | 20210622 |
一 摘要
本文主要介绍基于proxmox 虚拟机 直通显卡,最终希望在该宿主机上建两个虚机,各自直通一张gpu 卡,两台虚机同时使用。
二 环境信息
(一)物理机信息
品牌 | 型号 | 出厂配置 | 修改配置 |
---|---|---|---|
浪潮 | SA5212M5 | 42102/128G/SSD:240G2/SAS:2T 7.2K 2 /10G X7102/1G PHY卡*1/RAID卡 SAS3108 2GB | 去掉一张万兆光卡,插上两张英伟达 T4 |
GPU 卡信息
root@testgpu:/etc# lspci -nnk | grep "NVIDIA"
3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
root@testgpu:/etc#
(二) 软件信息
proxmox 6.4.4
三 实施
Intel 机器和AMD 机器 配置文件等相关命令是有区别的。
(一) Intel 机器 gpu passthrough
3.1.1 检查宿主机gpu 卡信息
经检查 该机器有两张nvidia T4
root@testgpu:/etc# lspci -nnk | grep "NVIDIA"
3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
root@testgpu:/etc#
3.1.2 Enable the IOMMU
3.1.2.1 编辑/etc/default/grub
先备份
root@testgpu:/etc# cp /etc/default/grub /etc/default/grub.bak.orig
root@testgpu:/etc#
修改
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream video=vesafb:off video=efifb:off"
修改说明:
pcie_acs_override=downstream 用于拆分iommu,pcie 一二槽同时插上时会共享带宽,如果不拆分iommu,当两个设备同时直通,其中一个就会显示占用
video=vesafb:off video=efifb:off :禁止启动和vesa驱动和efi启动的显卡
3.1.2.2 update-grub && reboot
root@testgpu:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.106-1-pve
Found initrd image: /boot/initrd.img-5.4.106-1-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
root@testgpu:~#
root@testgpu:~# reboot
3.1.2.3 Verify IOMMU is enabled
root@testgpu:~# dmesg | grep -e DMAR -e IOMMU
There should be a line that looks like “DMAR: IOMMU enabled”. If there is no output, something is wrong.
root@testgpu:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.009867] ACPI: DMAR 0x000000006AA82C08 0002A0 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.527674] DMAR: IOMMU enabled //找到这句
[ 0.902757] DMAR: Host address width 46
[ 0.902758] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
[ 0.902765] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902766] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[ 0.902770] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902771] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0
[ 0.902776] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902777] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 0.902780] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902781] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0
[ 0.902784] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902785] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0
[ 0.902788] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902789] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0
[ 0.902792] DMAR: dmar6: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902792] DMAR: DRHD base: 0x0000009d7fc000 flags: 0x1
[ 0.902795] DMAR: dmar7: reg_base_addr 9d7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.902796] DMAR: RMRR base: 0x0000006d1ea000 end: 0x0000006d1fafff
[ 0.902797] DMAR: ATSR flags: 0x0
[ 0.902798] DMAR: ATSR flags: 0x0
如果该命令没有输出,可能是VT-d 没有开启,需要到bios 里设置下。
3.1.3 Required Modules
修订 /etc/modules
先备份
root@testgpu:~# cp /etc/modules /etc/modules.bak.orig
新增如下内容
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
3.1.4 IOMMU Interrupt Remapping
root@testgpu:~# dmesg | grep 'remapping'
[ 0.902812] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.904778] DMAR-IR: Enabled IRQ remapping in x2apic mode //关键句
root@testgpu:~#
如果系统不支持interrupt remapping ,请配置
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
我这里是支持的。
3.1.5 Verify IOMMU Isolation
For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.
You should have something like:
root@testgpu:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/108/devices/0000:85:1e.5
/sys/kernel/iommu_groups/108/devices/0000:85:1e.3
/sys/kernel/iommu_groups/108/devices/0000:85:1e.1
/sys/kernel/iommu_groups/108/devices/0000:85:1e.6
/sys/kernel/iommu_groups/108/devices/0000:85:1e.4
/sys/kernel/iommu_groups/108/devices/0000:85:1e.2
/sys/kernel/iommu_groups/108/devices/0000:85:1e.0
/sys/kernel/iommu_groups/55/devices/0000:3a:0b.2
3.1.6 GPU Passthrough
3.1.6.1 HOST Configuration
宿主机需要禁用这张gpu卡 ,有两个方法
3.1.6.1.1 pass the device id 通vfion-pci
root@testgpu:~# lspci | grep NVIDIA
3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
af:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
root@testgpu:~# lspci -n -s 3b:00
3b:00.0 0302: 10de:1eb8 (rev a1)
root@testgpu:~# lspci -n -s af:00
af:00.0 0302: 10de:1eb8 (rev a1)
root@testgpu:~#
gpu 卡型号写到 /etc/modprobe.d/vfio.conf
root@testgpu:/etc/modprobe.d# ls
pve-blacklist.conf
root@testgpu:/etc/modprobe.d# less pve-blacklist.conf
[1]+ Stopped less pve-blacklist.conf
root@testgpu:/etc/modprobe.d# echo "options vfio-pci ids=10de:1eb8 " > /etc/modprobe.d/vfio.conf
root@testgpu:/etc/modprobe.d# pwd
/etc/modprobe.d
root@testgpu:/etc/modprobe.d#
3.1.6.1.2 禁用驱动
与3.1.6.1.1 二选一即可,不过我这次操作 是两步都做了
修改/etc/modprobe.d/pve-blacklist.conf
先备份
root@testgpu:/etc/modprobe.d# cp /etc/modprobe.d/pve-blacklist.conf /etc/modprobe.d/pve-blacklist.conf.bak.orig
root@testgpu:/etc/modprobe.d#
root@testgpu:/etc/modprobe.d# echo "blacklist radeon" >> /etc/modprobe.d/pve-blacklist.conf
root@testgpu:/etc/modprobe.d# echo "blacklist nouveau" >> /etc/modprobe.d/pve-blacklist.conf
root@testgpu:/etc/modprobe.d# echo "blacklist nvidia" >> /etc/modprobe.d/pve-blacklist.conf
3.1.6.1.3 更新配置 重启机器
root@testgpu:/etc/modprobe.d# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.4.106-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. // 这个错误直接忽略,因为proxmox 安装的debian 并不是一个完整的系统引起
root@testgpu:/etc/modprobe.d#
root@testgpu:/etc/modprobe.d# reboot
root@testgpu:~# lspci -nnk | grep NVI
3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Subsystem: NVIDIA Corporation TU104GL [Tesla T4] [10de:12a2]
root@testgpu:~#
3.1.6.2 VM Configuration
将设备传给vm
需要在设备宿主机上执行
qm set 100 -hostpci0 3b:00.0