Setting up Red Hat 5.4 Xen for SR-IOV using the Intel 82576 GbE

转载自：

https://communities.intel.com/community/wired/blog/2010/03/01/setting-up-red-hat-54-xen-for-sr-iov-using-the-intel-82576-gbe

Setting up Red Hat 5.4 Xen for SR-IOV using the Intel 82576 GbE

发贴人 Patrick_Kutch 于2010-3-1 18:53:54在Wired Ethernet

So Intel has this new super-cool SR-IOV technology that allows you to directly assign multiple VMs to an Intel NIC based on the Intel® 82576 GbE Controller.  But how, you may be wondering do you get it to work?  I’ve written a document that explains this, and am going to make a quick blog out of it.  I hope you find it useful, if you do, then my name is Patrick, if you don’t, then my name is Paul.:-)

 

It should be noted that there is a bug in the version of the Xen kernel that shipped with Red Hat* 5.4 Xen that causes the SR-IOV performance to be poor.  The upcoming Red Hat 5.5 release will have the fix in it.

1  Introduction

This document is intended to help those wishing to setup and configure SR-IOV on Red Hat 5.4 Xen with the Intel® 82576 GbE controller.

Included within these instructions are step-by-step procedures for installing Red Hat 5.4 Xen along with configuring the installation for SR-IOV usage and how to assign a SR-IOV Virtual Function to a Virtual Machine.

2  Server Requirements

The server where Red Hat 5.4 Xen is to be installed must SR-IOV.  There are a number of platform requirements for SR-IOV that include BIOS support and I/O re-mapping capability such as that available as part of the Intel® Virtualization Technology for Directed I/O (Intel® VT-d).

In addition a SR-IOV capable Ethernet controller is required.  The system used for the purposes of this paper has an Intel® 82576 GbE controller as the LOM, thus meeting the requirement for a SR-IOV capable Ethernet controller.

 

2.1 Server Used

At the time of the writing of this document there are very few commercially available servers that support SR-IOV.  The system used for this paper is a Supermicro X8DTN+ with the UEFI BIOS which supports SR-IOV.

In addition to the on-board Intel® 82576 GbE Controller, an Intel® 82572EI GbE Controller has been added to the server to be used for non SR-IOV traffic.

2.2 BIOS Configuration

There will be some sort of BIOS configuration required to enable SR-IOV for the server.  This section details the configuration required for the Supermicro X8DTN+.

2.2.1 Enable Intel® VT-d

This configuration is found under the ChipsetàNorth BridgeàTylersburg IOH ConfigurationàIntel VT for Directed I/O Configuration menu.

Figure 1 BIOS SR-IOV Configuration

2.2.2 Enable Intel® Virtualization Technology (VT-x)

In addition to enabling Intel® VT-d, the hypervisor itself also requires Intel VT-x to be enabled.  This is done through the BIOS option of:

Advanced->CPU Configuration->Intel Virtualization

Figure 2 Enable Intel VT-x

3  Installation of Red Hat 5.4 Xen

The first part of the installation is just like normal installation of a Red Hat distribution; simply put in the CD/DVD and boot up the computer.

3.1 Installation/Subscription Number

When prompted for the Installation Number dialog box, enter the appropriate Installation Number:

Figure 3 Installation Number

It is vital that a valid Installation/Subscription number be provided at this point.  If this is not done then the option to select Virtualization support (Xen) will not be available.

Continue with installation.

3.2 Selecting Virtualization Support

If a valid Installation/Subscription number was entered (see section 3.1) then a dialog should appear as follows:

Figure 4 Select Virtualization Support

If so, make sure Virtualization is selected and continue with installation as normal.

If, however the screen appears as:

Figure 5 Virtualization Option Not Available

This is an indication that a valid Installation/Subscription number was not entered as described in Section 3.1.

4  Configuration of Red Hat 5.4 Xen for SR-IOV Support

At this point Red Hat Xen 5.4 should be installed and running.  It will look just like ‘normal’ Red Hat Linux.  One difference will be the ability to launch the Virtual Machine Manager from the ApplicationàSystem Tools menu.

Additionally information about the Xen installation can be obtained from a command prompt by entering the command:

#xm info

4.1 Checking to see if any SR-IOV VF’s are Present

Now that Xen in installed and running, you may want to see if any SR-IOV Virtual Functions exist on the system (they won’t).  To verify this, enter the following command:

#lspci | grep Eth

The resulting output should look very similar to the following:

01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

This shows three Intel® Ethernet controllers, including the dual port Intel® 82576 GbE controller; however no Virtual Functions appear yet.

4.2 Modifying modprobe.conf

The modprobe.conf file, located in the /etc/ directory needs to be updated with some options for the Intel igb driver.  The additions to the file are as follows:

options igb max_vfs=7

blacklist igbvf

options loop max_loop=128

Make these modifications and save the file.

The resulting file I ended up with is as follows:

alias eth0 e1000e

alias eth1 igb

alias eth2 igb

alias scsi_hostadapter ahci

options igb max_vfs=7

blacklist igbvf

options loop max_loop=128

 

4.2.1 Explanation of Settings

·         options igb max_vfs=7

This option tells the igb driver to allocate sufficient SR-IOV resources for 7 Virtual Functions per port.

·         blacklist igbvf

The blacklist line makes it so modules ignore any igbvf devices that show up in the hypervisor PCIe space

·         options loop max_loop=128

By default Linux only has 7 loop devices.  SR-IOV requires many more, this option increases the number to 128

4.3 Modifying grub.conf

The grub.conf file presents options on how to load the kernel when booting.  This file is located in the /boot/grub/ directory.

This file needs to be updated to indicate that SR-IOV capability is desired.  This is accomplished by adding a single parameter to one of the lines in the grub.conf file.  And 2 additional parameters to another line.

Specifically “pci_pt_e820_access=on” needs to be added to the “module /vmlinuz-*” line.  While the kernel /xen.* line needs “iommu=1” and “msi=1” added.

My preference is to make 2 boot options, leaving the original one and making an alternate with SR-IOV enabled. To do this simply copy and paste the existing configuration to a new entry, rename it and then set the default to be the SR-IOV entry.  The resulting grub.conf file is as follows:

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /boot/, eg.

#          root (hd0,1)

#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00

#          initrd /initrd-version.img

#boot=/dev/sda

default=1

timeout=30

splashimage=(hd0,1)/grub/splash.xpm.gz

hiddenmenu

title Red Hat Enterprise Linux Server (2.6.18-164.el5xen)

       root (hd0,1)

       kernel /xen.gz-2.6.18-164.el5

       module /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet

       module /initrd-2.6.18-164.el5xen.img

 

title Red Hat Xen (2.6.18-164.el5xen)

       root (hd0,1)

       kernel /xen.gz-2.6.18-164.el5 iommu=1 msi=1

       module /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet pci_pt_e820_access=on

       module /initrd-2.6.18-164.el5xen.img

After doing this, the “Red Hat Xen (2.6.18-164el15xen)” option is default.  When the system boots next time, if you press a key at grub startup you will see something like Figure 6.

4.3.1 Explanation of Settings

·         default=1

o    This indicates that of the two boot options now in the grub.conf file, the default will be the newly created Red Hat Xen (2.6.18-164.el5xen) option.

·         iommu=1

This option tells the kernel to enable SR-IOV

·         msi=1

This option tells the kernel to enable MSI-X interrupts

·         pci_pt_e820_access=on

This option enables direct assignment of a PCIe device to a Virtual Machine

Figure 6 Grub Menu, with SR-IOV Option

4.4 Checking Again for SR-IOV VF’s

After making the changes outlined in Sections 4.2 and 4.3 reboot the server.  Now try once again to see if there are SR-IOV Virtual Functions available for assignment to a Virtual Machine.

To verify this, enter the following command:

#lspci | grep Eth

The resulting output should look very similar to the following:

01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

 

Note that as before, the two Intel® 82576 GbE controllers are still there, along with fourteen Virtual Functions.  Recall in Section 4.2 the options set 7 VF’s per port; with two ports that presents 14 Virtual Functions for assignment to a Virtual Machine.

If for some reason you do not see something similar, double check to make sure that the configuration steps in Sections 4.2 and 4.3 are correct – a simple typo will prevent things from working properly.

5  Assigning a Virtual Function to a Virtual Machine

This section will describe a mechanism for adding a SR-IOV VF to a Virtual Machine.  There are several ways to accomplish this - the explanation presented here will utilize the Virtual Machine Manager GUI provided with the Red Hat 5.4 Xen.

The steps outlined here will begin with the creation of the Virtual Machine.  If you wish to add a Virtual Function to an existing Virtual Machine, skip to step 5.2.

5.1 Creating a Virtual Machine

Run the Virtual Machine Manager application via the Applications->System Tools menu.

5.1.1 Step 1 – Start Create Wizard

Select the localhost in the window, and then the ‘New’ button.

Figure 7 Virtual Machine Manager

Follow the steps within the wizard to create a new virtual machine until you come to the point where the wizard prompts for information about the network.

5.1.2 Step 2 – Network Configuration

When the Virtual Machine creation wizard prompts for information about the Network to be used, simply go with the default of ‘Virtual Network’

Figure 8 Network Setup

Finish with the wizard.  When the Virtual Machine begins to boot up for the 1^st time, power it down using the Virtual Machine Manager.

5.2 Assigning the SR-IOV Virtual Function

Select the powered down Virtual Machine from the Virtual Machine Manager. 

5.2.1 Step 1 Hardware Configuration Tab

Then select the hardware tab.

If an emulated NIC already configured, it can be removed using this panel.

Figure 9 Removing Default NIC

  Step 2 – Add Hardware

Select the Add Hardware button at the bottom of the window.  The Adding Virtual Hardware window will appear:

Figure 10 Adding Virtual Hardware

Select “Physical Host Device” from the dropdown list, press the “Forward” Button”

5.2.2 Step 3 – Select 82576 Virtual Function

From the Device drop down list, scroll down until the 82576 Virtual Function devices appear and select one.

Figure 11 Select a Virtual Function

After selecting the Virtual Function, finish the wizard and boot up the Virtual Machine.  The new hardware will be discovered and depending upon the guest Operating System the driver will either installed automatically or you will be prompted for a driver CD.

Install the Intel® 82576 Virtual Function Driver and enjoy the SR-IOV functionality of the Intel® 82576 GbE Controller.

5.2.3 Virtual Function Number and Physical Port Relationship

Care should be taken when selecting the Virtual Function to be assigned to a Virtual Machine.  Each VF has a PCIe BUS:Device:Function ID’s.  As seen in Figure 11 the ID’s for the dual port Intel® 82576 GbE controller are numbered 01:10:n and 01:11:n. 

If the last number (n) is an even number, then the Virtual Function is on Physical Function (port) 0.  If the number (n) is odd, then the Virtual Function is on Physical Function (port) 1.

Note: Care should be taken not to assign the same Virtual Function number to more than one Virtual Machine.  The GUI will allow this to occur, however when the Virtual Machines are run, the first one loaded will function properly, however any subsequently loaded Virtual Machines that are assigned the same Virtual Function will immediately fail with an error as shown in Figure 12.

Figure 12 Attempt to Use the Same VF Twice