本文介绍了如何在Xen上使用Pass-through和SR-IOV,至于Pass-through和SR-IOV的原理可以查阅其他文章。
所谓Pass-through技术是指可以将PCI/PCIe设备直接分配给虚拟机使用,而SR-IOV设备除了有一个物理功能 (Physical Function, PF)之外,还可以许多虚拟功能 (Virtual Function, VF)给虚拟机使用。
本文使用的环境是:Ubuntu14 + Xen 4.6 + Intel 82599网卡
参考文献:
所谓Pass-through技术是指可以将PCI/PCIe设备直接分配给虚拟机使用,而SR-IOV设备除了有一个物理功能 (Physical Function, PF)之外,还可以许多虚拟功能 (Virtual Function, VF)给虚拟机使用。
本文使用的环境是:Ubuntu14 + Xen 4.6 + Intel 82599网卡
PCI Pass-through的使用
使用PCI Pass-through需要硬件的支持:- 主机需要支持Intel VT-d 或者 AMD IOMMU技术
- 在 Linux 内核需要启用 PCI Pass-through
root@ccrfox141:/home/ye# xl dmesg | grep -i vt-d | grep -i enable
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
也可以通过 "xl dmesg | less" 命令查看更多的信息。
要使用Pass-through,首先要找到设备的BDF (Bus, Device, Function),BDF由三个或者四个数字组成: DDDD:bb:dd.f,其中:
- DDDD is a 4-digit hex for the PCI domain. This is optional (if not included, it will be assumed to be 0000).
- bb is a 2-digit hex of the PCI bus number.
- dd is a 2-digit hex of the PCI device number.
- f is a 1-digit decimal of the PCI function number.
使用lspci命令可以很方便的找到设备的BDF:
root@ccrfox141:/home/ye# lspci | grep 82599
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
如果想将设备通过Pass-through的方式分DomU,可以在该DomU的配置文件上配置为:
pci = [ '05:00.0' ]
如果Linux内核开启了PCI Hot-Plug功能,可以直接分配给正在运行的VMs:
# xl pci-attach 05:00.0
但是在做这一步之前,需要先将要分配的设备绑定到pciback上,在Xen 4.2之前需要配置BootLoader等等繁琐的工作,Xen4.2和之后的版本提供了pci-assignable-*命令可以很方便的进行绑定pciback和分配Pass-through等操作。
- pci-assignable-add will first attempt to un-bind the device found at from its current driver (if any), and then attempt to bind it to pciback. If the device was bound to a driver, the identity of that driver will be stored, in case the admin wants to return it later. At this point, it is ready to be passed through to a guest.
- pci-assignable-list will list which devices are available to be assigned to guests. This will list all devices currently assigned to pciback, whether this was done by pci-assignable-add, or by the two methods mentioned in the previous section (linux command-line or manual sysfs commands).
- pci-assignable-remove will un-bind the device from pciback. If the -r option is specified, it will also attempt to rebind the device to original driver, allowing it to be used by domain 0 again.
简答来说,
pci-assignable-add命令会剥夺Dom0相应驱动对该设备的使用权,将其绑定到pciback后提供给DomU使用;pci-assignable-list可以列出所有可以被DomU使用的设备;pci-assignable-remove命令将设备归还给Dom0。这极大的简化了通过Pass-through分配设备的过程,只要知道了设备的BDF,就可以分配该设备给DomU,比如:
# xl pci-assignable-add 05:00.0
# xl pci-attach [domid] 05:00.0
使用完之后归还给Dom0:
# xl pci-detach [domid] 05:00.0
# xl pci-assignable-remove -r 05:00.0
如果想让设备开机自动绑定到pciback,可以修改/etc/default/grub文件,添加:
GRUB_CMDLINE_XEN_DEFAULT="xen-pciback.hide=(05:00.0)"
update-grub后查看/boot/grub/grub.cfg文件,可以看到xen-pciback.hide=(05:00.0)信息
使用SR-IOV
使用SR-IOV也需要硬件的支持,我使用的是Intel 82599网卡。
首先查看硬件是否被Linux检测到:
root@ccrfox141:/home/ye# lspci | grep 82599
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
以及查看该硬件的驱动:
root@ccrfox141:/home/ye# ethtool -i p5p1
driver: ixgbe
version: 4.2.1-k
firmware-version: 0x61bd0001
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
可以看到我的82599网卡的驱动是ixgbe,可以使用lsmod 命令查看ixgbe驱动是否工作正常:
root@ccrfox141:/home/ye# lsmod | grep ixgbe
ixgbevf 53248 0
ixgbe 274432 0
vxlan 45056 1 ixgbe
mdio 16384 1 ixgbe
dca 16384 2 igb,ixgbe
ptp 20480 3 igb,tg3,ixgbe
接下来就可以激活82599的虚拟功能(Virtual Functions)了,使用参数max_vfs 可以在加载模块的时候指定设备可以分配的最大虚拟功能(Virtual Functions)的数目。
首先移除ixgbe模块:
# modprobe -r ixgbe
重新加载ixgbe模块并使用max_vfs参数指定设备可以分配虚拟功能的最大数目:
# modprobe ixgbe max_vfs=7
注:设备能支持的最大虚拟功能数目可以到/sys/class/net/device_name/device/sriov_totalvfs文件中查看。
查看激活的虚拟功能:
root@ccrfox141:/home/ye# lspci | grep 82599
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
激活后的虚拟功能就可以通过Pass-through方式分配给DomU使用啦!
PCIback and the permissive option for PV guests
当给DomU分配地址空间较大时(大于4G),会出现PCI地址空间与系统地址空间冲突的问题:
[ 103.431754] xen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
[ 103.431754] You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
这是没有为PV guests配置许可选项的问题。有两种方法可以为DomU配置permissive
对该DomU开放所有设备的permissive,可以在该DomU的配置文件上添加:
#Changes the default value of 'permissive' for all PCI devices passed through to this VM
pci_permissive = 1
开放某个设备的permissive:
pci = [ '05:00.0,permissive=1' ]
也可以在添加时手动指定:
# xl pci-attach [domid] 05:00.0,permissive=1
参考文献: