| Xen 3.2 Bridged Networking Explained by Experiment |  |  |  |
| Written by Thaddeus Hogan |
| Saturday, 20 June 2009 12:01 |
| When I start leaning a new technology, I tinker with customizations to the defaults immediately because I learn more working out the bugs I create. When I recently started messing around with Xen on Debian 5 (Lenny) I found the network bridging setup to be rather confusing at first. There exists a lot of documentation explaining how to setup specific network configurations, but nothing that went through a complicated configuration in step-by-step detail. After a day of experimentation I figured out the exact symantics of the bridging configuration. This is still pretty non-advanced stuff, but I am attempting to explain it in detail to help make it clear to new users right from the start. DISCLAIMER: I am pretty new to Xen and am writing this from a beginner's standpoint. This isn't an attempt to explain complicated bridging setups, but rather to explain what is taking place at the various steps in the other examples I have found. BEFORE YOU READ THIS: Read the XenNetworking page at Xen Wiki. (http://wiki.xensource.com/xenwiki/XenNetworking ). Go through the whole thing, as it hammers out the basics of how networking in Xen works and will make the rest of this document make sense. What I am presenting here is an example based explanation of the concepts, but not concepts themselves. PLATFORM: This document is Debian oriented. Specifically I am using Debian 5 (Lenny) and Xen 3.2. It should apply to other Debian based distributions fairly well but I have not tested it. If you have the time and the knowledge, please consider porting this document to other distributions such as the RedHat, CentOS, Fedora family! (I probably will not be running Xen on those platforms anytime soon) INSTALLATION: I am assuming that you have successfully installed Xen and booted the Hypervisor and dom0. CONVENTIONS: I quote literal commands and file content if they are used within a sentence. The quotes are not part of the file content unless specified. Default ConfigurationThe default bridging configuration is defined in /etc/xen/xend-config.sxp using just the line "(network-script network-bridge)". If you boot dom0 with eth0 configured as usual and the default bridging line uncommented in xend-config.sxp then you should have network connectivity as defined for eth0 in /etc/network/interfaces. NOTE! The default xend-config.sxp states that the default bridge name is "xenbr0". This was not the case after I enabled the default bridging config. The physical interface eth0 was renamed to peth0 and the bridge was given the name eth0. This is not consistent with how bridging is explained in the XenNetworking wiki page or in the configuration file. From the config file: # Your default ethernet device is used as the outgoing interface, by default. Do an "ifconfig -a" and review the interfaces that now exist on the system. You should see what appears to be your original interface(s) and then several veth and vif interfaces: xen0:~# ifconfig -a What Exactly Happened?In Short When Xend started it did the following:
The Whole Story As it is explained in the XenNetworking page, a bridge called xenbr0 is created. The interface vif0.0 is added to the bridge and then veth0 is renamed to eth0. This was not the case with my Debian 5 / Xen 3.2 setup, so let's take a look at what we actually have: xen0:~# brctl show So, the bridge is called eth0, and it has the ip configuration that was specified for eth0 in the /etc/network/interfaces file: allow-hotplug eth0 How did the bridge get the IP information for eth0 exactly? It seems that xen runs the operating system's own "ifup" script to reconfigure the bridge with the ip information that the physical interface originally had. We can test this by commenting out the line "auto eth0" and rebooting. After having commented out the "auto eth0" line and rebooting, one would expect that the eth0 interface would not be up after the boot completes. However you will find that after Xend starts it comes up and is a bridge. So now we know that Xend runs the ifup command for the bridge interface after it renames the physical interface by prefixing it with a 'p' and creates the bridge. Renaming The BridgeMake sure you do this from a VGA or serial console because it will break your network connectivity. The next step for examining how the bridge works is renaming it. Edit the /etc/xen/xend-config.sxp file as follows: Change: (network-script network-bridge) Then reboot. After you get to the login prompt, you should see this error message from the boot sequence when Xend started: "Ignoring unknown interface abcd0=abcd0". Login and test for network connectivity, it should be broken. Run the route command, there should be no routes. Explanation Of Changes: The "bridge=abcd0" parameter specifies the name of the bridge interface. The "netdev=eth0" parameter specifies the name of the physical interface to use for the bridge. You must specify the netdev parameter because otherwise Xend will assume that the physical device name is the same as the bridge name and attempt to create pabcd0 as the physical interface. There will now be a bridge interface called abcd0. It will have the IP information copied from eth0 when it was renamed peth0, but it is not up. abcd0 Link encap:Ethernet HWaddr 00:1e:2a:cc:60:c5 Manually up the interface with the command "ifconfig abcd0 up". Test for connectivity again, it should be restored. Test against a local network host because there will not be a default gateway defined. Configuring The Bridge Properly At Boot Now that we have renamed the bridge and demonstrated that it does not get brought up or have routes for it added by Xen, we need to go about configuring the /etc/network/interfaces file to handle the ifup that Xend calls for the bridge. In /etc/network/interfaces, change the interface name from eth0 to abcd0 so that the section for that interface looks like this: iface abcd0 inet static Reboot the server and verify network connectivity. Why Rename The Bridge?First of all, I think that giving the bridge the same name as the physical interface is confusing, especially when you are first presented with all the new interfaces. Also when you want to add additional bridges, let's say to a dummy interface to create a private network, giving the bridges their own naming nomencalture helps maintain consistency when building bridges for different interface types. Secondly, the default configuation file's comments are not correct about the behavior of the scripts. Going through the exercise of renaming the bridge helps demonstrate how it actually works. Xen Virtual InterfacesSo far we have avoided the other half of the Xen networking equation, which are the virtual interfaces. The XenNetworking page describes an array of interconnected virtual interfaces. There are the vif* interfaces which are all present in dom0. Vif0.* is bound to the veth* interfaces in dom0, and vif1.*interfaces are bound to eth* in dom1, and so on. To demonstrate how these work in dom0 in a bridging configuration, I'm going to remove the IP configuration from the bridge then attach veth0 to the bridge for network connectivity. Update the /etc/network/interfaces file to bring up the bridge (abcd0) without an address. Then configure the interface veth0 to have your IP information with some extra commands to manage its vif0.0 counterpart. These changes aren't very intuitive so bear with me and I will explain it all below: iface veth0 inet static After making these changes, reboot the server. Make sure you are on a physical console because this will break networking again. Let's start with the configuration of the bridge, abcd0, at the bottom of the configuration above. The "up" line is a literal command to execute when ifup is called, and the "down" line is a literal command to call when ifdown is called. Running ifconfig and specifying an address of 0.0.0.0 removes an address if the interface at one point was configured with one, since downing the interface does not remove the address. The result of this configuration is that after a boot you should have the bridge interface UP and WITHOUT an address: xen0:~# ifconfig abcd0 According to the XenNetworking wiki page, the vif interfaces are where packets enter the Xen networking system, so this is where we will manipulate the connection to the physical network. The veth interfaces are attached to the vif interfaces and are the interfaces that the OS can use for network connectivity through Xen. In the configuration for interface veth0, there are a few lines that inject custom commands into the interface configuration process as performed by ifup and ifdown. Let's start with the line "pre-up ifconfig $IFACE hw ether 00:1e:2a:00:00:01". This assigns a MAC address to the veth0 interface BEFORE the interface is configured with an IP address and actually raised. After veth0 has a MAC address, we need to add it's counterpart vif0.0 to the bridge. This gets the packets from the bridge into the Xen networking system for delivery to veth0. The line "pre-up brctl addif abcd0 vif0.0" perfoms this task. At this point there are no more pre-up commands to execute, so ifup will configure the veth0 interface with the specified parameters (address, netmask, gateway) and then bring veth0 up. Now veth0 is configured and vif0.0 is in the bridge, BUT vif0.0 is still down. So after everything else is configured, which is what "post-up" means, we then can manually bring up the related vif0.0 interface with this line: "post-up ifconfig vif0.0 up". Since all of this is wrapped into the interface configuration, you should just be able to run the command "ifup veth0" to restore networking. Now lets look at our interfaces. The bridge will be in the same state, up and without an address. Interface vif0.0 should also be up and have no address: xen0:~# ifconfig vif0.0 Interface veth0 should be up and configured with an IP address: xen0:~# ifconfig veth0 And the bridge abcd0 should show that vif0.0 is a part of it now: xen0:~# brctl show Bring Up veth0 On Boot Finally, if you want to keep the network configuration this way, you'll need a way to raise veth0 on boot after Xend starts. So what better place than in a custom network-bridge script? Below is a script I created at /etc/xen/scripts/network-bridge-custom. It is similar to many multiple bridge configuration examples you will find, though it only specifies the one bridge (abcd0) at this time. I then tweaked it to bring up veth0 after it creates the bridge. #!/bin/sh Make sure that if you create a custom script like this that you give it the proper execute permissions. To use the new script, reference it in /etc/xen/xend-config.sxp instead of the default network-bridge script: Change: (network-script 'network-bridge bridge=abcd0 netdev=eth0') Now when Xend starts it will call the custom script and raise veth0 after configuring the bridge! ConclusionI hope that this helps explain bridged networking in Xen in a more thorough way than a regular howto or tutorial. This is not intended to be a practical application but instead an explanation of each part of Xen networking integration in regards to bridges. If you see any errors or can contribute clarifications, let me know! Finally, a fresh installation of Xen 3.2 on Debian 5 seems to have different default networking behavior than described in most other documentation I have read. Since I am just getting started with Xen I am going to assume that there were recent changes that obsoleted some documentation. Please feel free to copy this document or any part of it for other documentation if you find it useful. Also kindly give me the bump if you do! |
| Last Updated on Saturday, 20 June 2009 18:33 |
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
