Vsphere6.5 NVIDIA--Tesla--P40 GPU 直透

P40 GPU 直透

####pre
1.vsphere6.5 平台 + 4 张P40 GPU卡
2.guest os nvidia 驱动

###1.uiweb上设置 gpu 直通 模式

设置好后，需要重新引导系统，重启后，如上图，直通状态为 活动

###2. 命令行进主机系统，查看

[root@gpu52:/etc/vmware] cat /etc/vmware/esx.conf  | grep passthru
/device/00000:136:00.0/owner = "passthru"
/device/00000:005:00.0/owner = "passthru"
/device/00000:008:00.0/owner = "passthru"
/device/00000:131:00.0/owner = "passthru"

说明，4张gpu卡已设置直透了

###3. 新增2台虚拟机win2K12 和 test-3 （centos7.8）

###4.使用vcenter操作
#4.1

#4.2 选择虚拟机–右键–编辑设置–虚拟机硬件–添加PCI设备

#4.3 选择PCI设备—预留对等内存–

#4.4 虚拟机选项—高级 —编辑配置

新增
pciPassthru.use64bitMMIO   TRUE
pciPassthru.64bitMMIOSizeGB 64

##4.5

####5.虚机安装显卡驱动

#5.1 下载

####5.2 centos7 安装驱动

####查看显卡信息

[root@localhost ~]# lspci | grep -i NVIDIA
0b:00.0 3D controller: NVIDIA Corporation GP102GL [Tesla P40] (rev a1)

####安装依赖环境
yum install kernel-devel gcc -y


###.检查内核版本和源码版本

[root@localhost ~]# ls /boot | grep vmlinu
vmlinuz-0-rescue-07cf47a5080d46b8a3558c3c1979b0ec
vmlinuz-3.10.0-1127.el7.x86_64
[root@localhost ~]# 
[root@localhost ~]# rpm -aq | grep kernel-devel
kernel-devel-3.10.0-1127.19.1.el7.x86_64


###屏蔽系统自带的nouveau

[root@localhost ~]# lsmod | grep nouveau
nouveau              1898794  0 
video                  24538  1 nouveau
mxm_wmi                13021  1 nouveau
wmi                    21636  2 mxm_wmi,nouveau
i2c_algo_bit           13413  1 nouveau
drm_kms_helper        186531  2 nouveau,vmwgfx
ttm                    96673  2 nouveau,vmwgfx
drm                   456166  5 ttm,drm_kms_helper,nouveau,vmwgfx


修改 /lib/modprobe.d/dist-blacklist.conf 

将nvidiafb注释掉:
#blacklist nvidiafb 

然后添加以下语句：
blacklist nouveau
options nouveau modeset=0


####重建initramfs image

[root@localhost ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
[root@localhost ~]# 
[root@localhost ~]# dracut /boot/initramfs-$(uname -r).img $(uname -r)


####修改运行级别为文本模式

[root@localhost ~]# systemctl set-default multi-user.target
Removed symlink /etc/systemd/system/default.target.
Created symlink from /etc/systemd/system/default.target to /usr/lib/systemd/system/multi-user.target.

####，重启，

[root@localhost ~]# lsmod | grep nouveau                                                  
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# 


####上传 驱动


[root@localhost ~]# ls
anaconda-ks.cfg  NVIDIA-GRID-vSphere-6.5-430.99-432.44.zip
[root@localhost ~]# 
[root@localhost ~]# unzip -d /tmp NVIDIA-GRID-vSphere-6.5-430.99-432.44.zip 
Archive:  NVIDIA-GRID-vSphere-6.5-430.99-432.44.zip
  inflating: /tmp/430.99-432.44-grid-gpumodeswitch-user-guide.pdf  
  inflating: /tmp/430.99-432.44-grid-licensing-user-guide.pdf  
  inflating: /tmp/430.99-432.44-grid-software-quick-start-guide.pdf  
  inflating: /tmp/430.99-432.44-grid-vgpu-release-notes-vmware-vsphere.pdf  
  inflating: /tmp/430.99-432.44-grid-vgpu-user-guide.pdf  
  inflating: /tmp/430.99-432.44-whats-new-vgpu.pdf  
  inflating: /tmp/432.44_grid_win10_server2016_server2019_64bit_international.exe  
  inflating: /tmp/432.44_grid_win7_win8_server2008R2_server2012R2_64bit_international.exe  
  inflating: /tmp/NVD.NVIDIA_bootbank_NVIDIA-VMware_430.99-1OEM.650.0.0.4598673-offline_bundle-16285628.zip  
  inflating: /tmp/NVIDIA-Linux-x86_64-430.99-grid.run  
  inflating: /tmp/NVIDIA-VMware-430.99-1OEM.650.0.0.4598673.x86_64.vib  
[root@localhost ~]# cd /tmp/
[root@localhost tmp]# chmod +x NVIDIA-Linux-x86_64-430.99-grid.run 
[root@localhost tmp]# ./NVIDIA-Linux-x86_64-430.99-grid.run  --kernel-source-path=/usr/src/kernels/3.10.0-1127.19.1.el7.x86_64 -k $(uname -r)
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 430.99.....................................................................................................................................................................................................................................................................................................................................................................................................................................................
[root@localhost ~]# nvidia-smi
Wed Aug 26 02:20:19 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.99       Driver Version: 430.99       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P40           On   | 00000000:13:00.0 Off |                  Off |
| N/A   28C    P8    10W / 250W |      0MiB / 24451MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

至此，透传成功！