How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

2011-04-29 13:17

Setting up Xen4 has been tricky, because of the move to the latest Linux kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades. Important note: I couldn’t get Xen 4.0.1 release work on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel available from http://xenbits.xen.org/xen-4.0-testing.hg and http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre and 2.6.32.21 as of this writing). This may or may not be a problem for you. Install prerequisite packages yum -y groupinstall "Development Libraries" yum -y groupinstall "Development Tools" yum -y installtransfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial glibc-devel Install rpmforge repository. Follow the instructions on CentOS Wiki. wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm yum check-update Install mercurial and git yum -y install mercurial git Compile Xen hg clone http://xenbits.xen.org/xen-4.0-testing.hg cd xen-4.0-testing.hg make -j16 xen make -j16 tools make -j16 stubdom Compile pvops kernel make kernels make -j16 dist This is the most critical step, and if you don’t configure the kernel correctly, booting will fail badly. make kernels builds the kernel with the default .config. To modify it, change to build-xxx directory and run make menuconfig. Make sure to have the following in your config file # These are important for CentOS/RHEL CONFIG_SYSFS_DEPRECATED=y CONFIG_SYSFS_DEPRECATED_V2=y CONFIG_ACPI_PROCFS=y CONFIG_XEN=y CONFIG_XEN_MAX_DOMAIN_MEMORY=32 CONFIG_XEN_SAVE_RESTORE=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PCI=y CONFIG_PCI_XEN=y CONFIG_XEN_BLKDEV_FRONTEND=m CONFIG_NETXEN_NIC=m CONFIG_XEN_NETDEV_FRONTEND=m CONFIG_XEN_KBDDEV_FRONTEND=m CONFIG_HVC_XEN=y CONFIG_XEN_FBDEV_FRONTEND=m CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XEN_BACKEND=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_XENBUS_FRONTEND=m CONFIG_XEN_PCIDEV_FRONTEND=y If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as modules, use the following in your config. CONFIG_XEN_DEV_EVTCHN=m CONFIG_XEN_BACKEND=y CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_XENFS=m Install Xen and kernel make install-xen make install-tools PYTHON_PREFIX_ARG= cd build-* make modules_install make install For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems, this is not required. Create initrd (above make install usually takes care of creating an initrd matching your configuration, but just in case) depmod 2.6.32.21 mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21 Update grub title Xen 4 + 2.6.32.21     root   (hd0,0)     kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all     module  /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0 console=ttyS0,115200n8     module  /initrd-2.6.32.21.img Add the following to /etc/modprobe.conf to load all the Xen modules install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback; /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true OR, add the following to /etc/rc.local (I had trouble making the modprobe work) if( uname-r | grep"2.6.32.21") ; then    formod inxen-evtchn xen-netback xen-blkback xenfs; do         /sbin/modprobe $mod    done    /etc/init.d/xend start fi Add xenfs to /etc/fstab none /proc/xen xenfs defaults 0 0 Test it echo "savedefault --default=X --once"| grub --batch (where X is the entry forXen) reboot /etc/init.d/xend start xm list Troubleshooting The biggest problem will be kernel not booting properly. There might be various reasons. Make sure to use the right set of configuration options for your hardware Make sure to have all Xen options correctly compiled out_of_memory kernel OOPS CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number Add more memory to dom0 with dom0_mem=4096M Network card not appearing – make sure to have the right network module (bnx2x for BroadCom NetXtreme II card) inserted Missing /dev/xen entries. Means that you haven’t loaded the xen-evtchn and xen-gntdev. Run modprobe xen-evtchn xen-gntdev See step 9 above to update /etc/modprobe.conf If you see the following in /var/log/messages xenconsoled: Failed to contact xenstore (Connection refused) Do step 2 to insert all the appropriate modules, and verify that you can start xend Installing VMs First, verify that the initrd includes all the necessary Xen frontend modules. This is not necessary for hvm VMs, but for paravirtualization it’s crucial to have all the right netfront and blkfront modules installed. A sample mkinitrd for domU is below mkinitrd -v-f /boot/initrd-2.6.32.21-domU.img  --with=blktap \         --with=xen-blkback --with=xenfs 2.6.32.21 Install virt-manager yum installvirt-manager Configure /etc/xen/xend-config.sxp to enable access through libvirt and virt-manager. Add the following lines (xend-http-server yes) (xend-unix-server yes) (xend-port        8000) (xend-relocation-port 8002) Restart xend /etc/init.d/xend restart Start virt-manager as root Install a VM as usual Referenceshttp://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most authoritative reference for compiling the pvops kernel. http://wiki.xensource.com/xenwiki/Xen4.0 http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ (this is Ubuntu specific, but has some useful details)