Xen Basics

Copyright © 2006 Marcel Koßin
You can use this tutorial under the terms of the GNU Free Documentation License
Feedback, ideas? Please send mail to feedback@nodemaster.de
The most current version of this tutorial can be found at http://www.nodemaster.de/xen-basics/

Abstract

Virtualization is no new technique. It is often used for large scale servers, but it is relatively new for x86-based servers. In classical data centers, based on intel compatible hardware, a server often is idle, waiting for a peak workload. It would be desirable to use the computing power for other things, but this is not possible in most cases.

After following the provided steps in this document you will end up with a Xen server with at least one Xen domU. You will be accustomed to the terminology used in XEN environments and know how to achieve the following tasks:

• Install Xen with Linux as dom0
  
• Create unprivileged domains
• Do basic administrative tasks
  

Requirements

Although this tutorial provides the nessecary steps to get a simple Xen installation up and running you need some basic knowledge about other things:

• Good skills in setting up and administrating a Linux system
  
• Good networking skills
• Basic understanding of the Linux kernel
• Basic knowledge of LVM
  

Of course, it's not possible to do the practice in theory. Therefore you will need a "bunch of ICs thrown into a big metal box", also known as computer ;-) It should meet at least the following requirements:

• x86-compatible
• at least 256MiB RAM. Of course, as always: "More is better.", "Much more is much better." (In fact you can do this on machines with less RAM, but it won't be fun.)
• 5GiB of spare disk space additional to the disk space you need to install your favourite distribution.
• Linux supported network card
  

The technique at a glance

If you are new to XEN, or even virtualizaton, you need to understand some basics. Otherwise this topic might be too confusing. Please take the time to read this topic, even if you are familiar with an other virtualization technique. It might save you a lot of headaches.

Please keep in mind that this tutorial is x86-centric. Especially the processor model can't be transfered to any other architecture. At the moment Xen supports PowerPC processors as well. Other architectures will be adopted soon, but the following assumptions nessecarily won't apply to them.

If you examine a standard operating system installed on intel-compatible hardware you will find, that the operating system is designed to gain exclusive access to the underlying hardware. After loading the operating system, the BIOS passes the access to it. While booting, it initializes the hardware, and makes use of specific drivers to control it. At no point the operating system bargains for anything other, that might access the hardware. In order to let more than one operating system access the hardware, we need to virtualize the hardware. In fact we need something, that controls access to the hardware, and takes care, that there is no concurrent access, which might influence each other, in an undesired way.

The Xen developers did that for us. Xen is a modified Linux kernel, which will be booted by the bootloader, initialize the hardware and will subsequently boot an other operating system. This new intermediate layer, called “Virtual Machine Monitor” or “Hypervisor” now controls the access to the hardware and virtualizes it to the operating system running on top of this Hypervisor. The virtualized Hardware, which can boot an other operating system is called “domain” in Xen terminology. The first started domain – often refered to as dom0 – has some special privileges. In a default configuration this domain controls most of the Hardware and can access it directly. In dom0 you will do most of the administrative tasks, that need to be performed, to start and administrate further virtualized machines. Every other domain is refered to as “unprivileged domain” or, in short, “domU”. These domains have normally no direct access to the hardware.

Rings of Power

To achieve such a solution we need to have a basic understanding of the architecture of x86- and x64-processors. I will refer only to the "x86-processor", since x64 is the successor of x86 and as it is backward compatible, most statements for x86 are also valid for x64. To realize, how virtualization is done on a processor we have to exercise some history. With the 1982 released, 80286 a completely new concept, was introduced to Intel Processors. The Intel processor could now be switched into a so called, "protected mode".  Besides a couple of other advantages it was now possible to execute code in different "rings", often also refered to as "priority levels" of the processor. Four rings are available, numbered from zero to three. Ring zero is allowed to execute any operation on the CPU, while the other rings permissions are handled the more restrictive, the more their number ascends, with ring three as the most restrictive ring. The rings communicate with each other using gates. Thus ring three is able to contact ring zero, requesting operations, which are only possible in kernel space. Ring zero validates, if the request is valid and authorized and executes it, returning the desired data. As a matter of fact, that's the reason, why user space programms shouldn't be able to crash a complete system.

While in theory it is possible to use four rings to deescalate privileges, in reality most operating systems only use two. These rings are often refered to as "user mode" and "kernel mode". Those are running on ring three and ring zero of this model, with user mode running in the most unprivileged ring and kernel mode having access to all functions of the processor. The fact, that the other rings are neglected is mainly based on maintaining a good portability of the operating system. Many other processor architectures like RISC for example only provides two rings. Thus it would be complicated to port an operating system, relaying on more than two priority levels, to an architecture supporting only two.

Software based Xen approach

As we now know, in theory, we have all we need to do virtualization on x86-based hardware. It should be possible to run a hypervisor in ring zero, starting many Operating Systems in ring one, and letting them execute their user space programms in ring three. Sounds very easy in theory, but in reality it is nessecary to convert the operating system, to be able to run in ring one, instead of ring zero. This is not easy, without access to the source code.

The Xen developers chose Linux for this purpose. Since it is Open Source, ist is very easy to implement new techniques. They modified the kernel to run in ring one, and also used the Linux kernel with special modifications as Hypervisor, running in ring zero. This kind of virtualization is called para-virtualization. Adopting such changes to other Operating Systems, to allow them to run as a Xen guest should be relatively easy. Especially Open Source Operating Systems can adopt these changes very quickly. For closed Source Operating Systems, such as Microsoft Windows, this is not true, since changes cannot be done by the Community. But rumors are afloat, that a portation of Windows XP has been completed, but mustn't be published due to a NDA.

As the operating system is aware of its virtualized hardware, it's possible to create and use optimized drivers. It's possible to use any block device as harddisk in domU's. Xen only does the conversion of domU's hda1 to dom0's sdg12 as an example. Source code for the virtual NIC Driver is reasonably small, since it's data never hits a real cable, until it's packages passes a physical NIC e. g. in Dom0. In the latter case the real NIC takes care about the underlying copper cable and proper access to it. This results in less overhead compared to emulated hardware.

Hardware based Xen approach

Since it might not be possible to adapt every operating system, due to several reasons the hardware vendors answered with virtualization techniques soldered to their CPUs. AMD calles it Pacifica, while Intel names it VMX. With this approach there is an additional layer introduced underneath the "priority levels". You can think of it as "ring -1".

The advantage is obvious. The hypervisor can run on this newly introduced ring, while the Operating Systems lying on top, are able to run unmodified in ring zero. From their point of view, they are running on standard x86-based Hardware and know nothing about the underlying virtualization. Xen supports this mode of operation and is able to run, for example, an unmodified Windows as an unprivileged domain. The disadvantage is, that you need very up-to-date hardware, which supports this kind of virtualization.

Getting our hands dirty

Due to lack of newer hardware, I will now guide you through a Xen installation using the software based approach. I'll add information for the hardware based approach, as soon as I get hands on appropriate hardware at home.

If you'd like to install Xen as suggested in this tutorial, please make sure, that you don't rely on SCSI Hardware or other special things, not nessecarily found in commodity hardware. Unfortunately, there are many drivers, that are not included in the tarball provided by Xen Source. For that reason you should use this tutorial only as starting point. It will provide a quick start on commodity hardware, but I encourage you to compile your own kernel later on.

Preparing dom0

To prepare a dom0, you need to install a standard Linux distribution. Choose your flavour. Whatever distribution you like to use is negligible. We will use a generic approach to get things up and running. Of course you can use appropriate packages provided by your distributor, if you like, too. In either case just make sure, you have done these things:

• Choose grub as bootloader. You have other alternatives, but none of them is called lilo! This tutorial assumes you use grub.
  
• The distribution should be compatible to, and well tested with 2.6-Linux kernels and must support hotplug/udev (saves you from headaches, trust me).
• To make things easy, make sure you have a spare partition, a logical volume of at least 5GiB is prefered (this should be sufficient for two small domains with swap). If you are not using LVM just leave an empty partition with enough disk space. We will create the LVM later on.
• LVM requires lvm2 installed ;-)
  
• Make sure you need no proprietary drivers, this applies to many hardware accelerated graphic adapters. They won't work in most cases (In fact you can get them up and running, but it's a pain in the ass). Please use the non-accelerated, free drivers, if available.
• You need bridge-utils and iproute package installed, if you'd like to connect your Xen domains to dom0 or the internet.
  

With this things in mind, go ahead and install your distribution as you are used to. Install all packages you might need, to administrate it. If you are ready, make sure, that this system works as espected. Many failures that emerge in Xen are unfixed in this state of the installation procedure, leading to wrong conclusions and making debugging hard. Thus double check if everything works. Later on, if you get in troubles with your Xen dom0, reboot your machine, choosing a standard kernel and check if the problem persists.

Installing Xen

Point your Webbrowser to the download area of xensource.com and download the 32 bit SMP Tarball. Unpack it using tar.

# tar xzf xen-3.0.3_0-install-x86_32.tgz

Change into the new directory. At this point, you want to read the README file. It contains useful information about the Installation procedure. After completing that task, really read it! It helps you understand, what we are doing now. Subsequently install Xen as recommended in the README.

# cd dist
# sh ./install.sh

After this procedure you have to configure grub to boot Xen and afterwards an initial domain - dom0. If your distribution needs an initrd, then you have to specify it using the grub "module" directive, rather than the "initrd" directive. Please keep that in mind, since it is a typical beginner's mistake. And you don't want to count yourself to them, right? ;-) OK, now lets have a look at the menu.lst of grub:

1| title           Debian, Xen 3
2| root            (hd0,1)
3| kernel          /boot/xen-3.0.gz
4| module          /boot/vmlinuz-2.6-xen root=/dev/hda2 ro
5| savedefault
6| boot

This should look quite familiar, to the default entry, your distribution created for you, after installing itself. The first 2 lines are quite self explanatory. At line three, it is getting interesting. Instead of the kernel of your distribution you now need to boot the Xen kernel, which is in fact a customized Linux kernel. This is your hypervisor and will take care of managing the resources for the Xen domains. Line four shows the kernel of the first domain (dom0). Notice, that the grub directive "module" is prepended instead of "kernel". If you need an additional initrd, you have to make sure, that "module" is prepended, instead of the directive "initrd".

The example above is a save default. You can copy it to your menu.lst, only changing the "root" directive and altering the root kernel command line appropriate to your needs. You can, in theory, use the kernel and initrd shipped with your distribution. But in order to work correctly the kernel of your distribution needs to support the Xen extensions. Some distributors have activated this features for you, some have not. Thus your mileage may vary. This tutorial assumes, you installed the Xen standard kernel, with its initrd.

Activating needed services at boot time

There are two things that need to be done, in order to get Xen up and running after booting the machine. First of all, you have to start "xend". This daemon takes care of the communication between the Hypervisor and your domains. It's possible to manually start and stop it, like any other init-driven daemon with it's script located at "/etc/init.d/xend". But the Xen install-script also installed an other script "/etc/init.d/xendomains". The latter will take care of booting your domains at machine startup, and also will gracefully shut down every running domain, if your machine is going down for halt or reboot. In order to execute this scripts while booting dom0, you have to add them to the appropriate runlevel. This can be achieved by using distribution dependent scripts:

Adding services using update-rc.d on Debian or Ubuntu
# update-rc.d xend defaults
# update-rc.d xendomains defaults

Adding services using insserv on SuSE
# insserv xend
# insserv xendomains

The TLS library

There is few software, that performs very poorly, carrying out certain operations under Xen. One of them is the glibc TLS (Thread-local Storage, not Transport Layer Security) library. To avoid a penalty of up to fifty percent, or even more you should move it, in order to avoid it's usage. You should repeat the following step on every domain, including dom0 and all domU's.

mv /lib/tls /lib/tls.disabled

Now it's time to do a reboot. Don't forget to choose the correct entry from grub's boot menu, the next time. :-D

Welcome to dom0

After the first boot, lets check, if everything is running as expected. You are now located within the dom0. You have a bunch of tools, which have been installed, while setting up the Xen installation. The tool, you will use most of the time is called "xm". With this neat tool you can administrate your domU's. The option "list" of xm shows all running Xen domains. Lets try it:

root@cassini:~# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      500     1 r-----   268.0

There is one domain running. It is called Domain-0 and has the id zero. It uses 500MiB of RAM, has access to 1 virtual CPU and has been running for 268.0 seconds (which is, in fact, the total computing time, rather than the uptime). If you see an output similar to that, Xen is up and running. Congratulations!

Installing a domU

After booting Xen and verifiying, that dom0 is up and running correctly, it is now time to build the first domU. There are many possibilities for creating domU's. Especially the choice, regarding the blockdevice backend, on which the domU will be installed, is huge. In this tutorial we will use LVM as backend, since it has a couple of advantages while tinkering with Xen.

• Simple creation, deletion and resizing of virtual harddisks
• Simple mounting in dom0 for administrative tasks
• Quick backup / rollback using LVM snapshots
  

Preparing a Partition for use with LVM

You can skip this part safely, if you are already using LVM. If not we will now prepare the spare harddisk partition for use with LVM. If you are not familiar with LVM, please refer to the Linux LVM Howto in order to get a deeper insight of how LVM is working. I assume you have a spare and empty partition /dev/hda5, which can now be used for our purpose. Lets start preparing a physical volume and creating a volume group.

# pvcreate /dev/hda5
# vgcreate xendisks /dev/hda5

Preparing a logical volume for our first domU

Now we will create the virtual harddiskss On one of both, we will install the operating system of our domU. The other one will be utilized as swap. Both have to be formated with an aproproate filesystem. I prefer ext3 for my filesystems, but it's OK to choose an other one here. Although, regarding the creation of the swap space, you have not that many choices :-)

# lvcreate -n vm01-swap -L128M xendisks
# lvcreate -n vm01-root -L2G xendisks
# mkswap /dev/xendisks/vm01-swap
# mke2fs -j /dev/xendisks/vm01-root

Now simply mount the root disk and install any linux distribution on it you like to use. To achieve this, most distributions provide tools to install (bootstrap) a system, into a directory of your choice. Here is a short list, which is not meant to be complete:

• Debian and Ubuntu provide "debootstrap".
• SuSE Linux has an option "Install into directory" in YaST.
  

Create a config for your domU and boot it

Create a file called "/etc/xen/vm01", and open it in your favourite editor. We will put the most important information there, which are needed to boot our system. As you can see below we use the same kernel and the same initrd for our domU as we use for our dom0. It is possible to choose an other kernel, and even to use a kernel on the domU's blockdevice, but we won't address this issue in this tutorial.

Line three and four should be self explanatory, too. Choose how much RAM (in MiB) will be available to your domU, and how the domain should be named. The name has to be unique. Keep that in mind! Except for dom0 the domain id will increment with every started domain and with every reboot of a domU. Thus, if you reboot a domU, it will get a new id. The only way to correctly identify the domU is its name.

1| kernel  = "/boot/vmlinuz-2.6.16"
2| ramdisk = "/boot/initrd.img-2.6.16"
3| memory  = 128
4| name    = "vm01"
5| disk    = [ 'phy:xendisks/vm01-root,hda1,w', 'phy:xendisks/vm01-swap,hda2,w' ] 
6| root    = "/dev/hda1"

Line five is interesting. Here the virtual harddisks for the domU are defined. The syntax is pretty simple. We provide two partitions to the domU (/dev/hda1 and /dev/hda2). Both are writable for the domU (notice the "w"-flag at the end). "phy" means, we use a physical blockdevice. As afore mentioned, there are other alternatives, which are not discussed in this tutorial. After the "phy"-statement we provide the path to the blockdevice without prepended "/dev/".

The last line should be well known from the linux kernel. It just tells the domU which device to mount as root partition. After doing that make sure, that you reflect the virtual disk layout in the fstab of your domU. Now it's time to boot the domU. Make sure all logical volumes of this domU are unmounted in all other domains (including dom0). Otherwise Xen refuses to use them. Start the domU with the following command:

# xm create /etc/xen/vm01 -c

The option "create" starts the domU specified in the configfile, whose path has ben provided. With "-c" xm attaches to the serial console of the started domU. Without this option xm intializes the domU, and then gives back the control to the shell. Anyway, now you should see something very familiar. It looks like Linux booting on a computer. :-D If everything went fine, your domU boots and provides a login prompt afterwards. Have Fun!

Networking in Xen

After doing a default install, as described above, the domains now should run in bridged mode. Thus, if a domU needs to connect to the network, it simply needs an IP-address and a proper IP configuration. Use the favourite system setup tool, provided by your distributor to achieve that. The domU now should be able to connect to every other machine connected to eth0 of the dom0. To get a better understanding how brided networking works in Xen, at first let's have a look at the topology of the bridges in dom0:

# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.feffffffffff       no              vif0.0
                                                        peth0
                                                        vif1.0
                                                        vif2.0
                                                        vif3.0

Xen created a bridge called xenbr0. This bridge consists of several interfaces with fancy names. The interface called peth0 is the physical (that's, what the "p" stands for) interface eth0. The other interfaces are "virtual interfaces". They are the "connectors" to the domains network cards. Think of them as ports on a switch, if you like. The first number, after vif tells, which domain it belongs to, the number behind the dot represents the information, to which interface at the appropriate domain it's connected to. Thus vif1.0 is connected to interface eth0 of domain1. The second interface eth1 of domain1 would be connected to vif1.1 at xenbr0 in dom0.

As seen in the table, there is also a vif0.0. This one is connected to eth0 on dom0. At boot time Xen creates this virtual interface, and then connects eth0 of dom0 to it. The physical eth0 is then called peth0, and gets connected to the bridge, too. In fact, it is a little bit more complicated, to get this up and running. But this should be enough information, to provide an insight on how bridged networking is done on Xen. You should keep this in mind, and refer to it, you start tinkering with iptables, or anything similar on your domains, since this is beyond the scope of this tutorial.

Basic administrative tasks

This tutorial will explain only the most important options for xm. If you would like a deeper introduction please refer to the man-page, or to one of the sources, provided at the end of this tutorial. At this point you already know the options "list" and "create". With "create" a domU is started, with the parameters of the given config file:

# xm create /path/to/configfile [-c]

The option "list" shows the status of all running domains. For domain specific operations you need either the name of the domain or the id. Both values are displaxed by "xm list". But, as said before, be careful using the id, since it changes during reboots.

# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      500     1 r-----   268.0
vm01                               1      128     1 r-----    18.2
vm02                               2      128     1 r-----    12.9

The other options, often needed are "reboot" and "shutdown". Both options need either a domain name or it's id as an argument. If you shutdown or reboot a domain via xm it tries to do this graceful, which means, it tries to switch to runlevel zero, if possible. Afterwards a reboot is enforced if it was requested.

# xm shutdown [domID|domName]
# xm reboot [domID|domName]

Things you shouldn't expect

First of all, Xen will never be as fast as doing the same stuff on separate mashines. This should be obviuous, but there are people who believe, Xen is the answer to every problem.  But this is in fact not true. If you have many mashines, often idling around doing nothing else than writing "-- MARK --"'s to syslog, then your chances are good, that you can set them up into a domU. But if your mashine is often under heavy and unpredictable load, then chances are good to screw up the Xen node if installed as domU. There are many limitations regarding the physical hardware, which have to be considered.

System I/O

Xen domains causing a lot of system I/O, especially on HDD's will impact all Xen domains. Of course you can minimize this effect by using more or faster HDD Controllers, but keep in mind, that the conversion of domU's "virtual block devices" to real blockdevices is done by dom0. Thus if it's I/O-scheduler is fed up, all domU's will idle wait for data. You should avoid applications on Xen which are doing a lot of I/O.

Video driver

If you need graphical access to you domU, you only can use a framebuffer. Thus if you need to run applications which needs graphic acceleration, than you probably don't want to use it on Xen. I've heard of some solutions, that export a second graphic adapter to a domU, where you can use it with a native driver. I never tried this so far, since this refered only to hardware virtualization and leads to the following problem:

Direct access to hardware

It is possible to export any PCI card (this includes AGP and PCI-X) to a domU. Thus it is not accessible in dom0, but domU can access it with a native driver. As you can imagine you'll always won't have enough slots for all the hardware you need ;-)

Power Management

Xen is most interesting on servers, but if you are planning to use it on your laptop, you should be aware of the fact, that Xen doesn't support power management, and possibly would never do. In non-Xen universe the operating system is responsible to decide, whether for example a HDD should be spinned down. This is of course no longer possible, since the disks are virtualized and other domU's are perhaps running on the same disk. As a matter of fact Power Management has to be done by Xen in this scenario. But Xen is not aware of the state of the Operating System. Thus it's hard to decide wether or not it's possible to spin down the disk.

Other things people expect

There are some limitations, that are not only valid for Xen, but also for normal Operating Systems, but for some odd reasons people hope they might work on Xen since it virtualizes hardware.

Dynamic memory allocation. This won't work for most operating system without Xen either. I guess there is confusion because in the manual there is a command, that increases or decreases the memory of a running domU. It's true, that it is possible to increase the memory on the fly, but the Operating System needs a reboot to allocate it afterwards. But this seems to be an Operating System issue to me, rather than a Xen issue, since the Linux kernel is not designed to allocate additional RAM during runtime.

Mounting HDD's in dom0 and domU's. It's not possible to mount a harddisk in dom0 and domU at the same time (in fact you can do it, but it will screw up your data). This again is no Xen issue. It's an Operating System issue. If you need to have access to filesystem from within both domains you need shared storage.

Some advice. If you build a Xen node always keep in mind, that if you boot the hardware, there is a special order how domU's are started. It's always a bad idea to put services you rely on like DNS, NFS, syslog-Server, or anything other, into a domU. It seems xend starts the domU's first time in alphabetical order of their configuration filename. It shuts them down, starting with the highest id. If you reboot the first domU (let's assume it does DNS) it will be shut down at first. This might break things like unmounting NFS shares on other domU's. If xend starts it begins to start all automatically stopped domU's in reversed order. After that it starts the domains in /etc/xen/auto/. I can't stress this enough: Don't put vital services on domU's (better not to put it on dom0, too). Keep in mind, what services you'll loose in case of a hardware failure.

Other resources

Xen Wiki

Archive of mailing list xen-users