Configuring VXLAN and GRE Tunnels on OpenvSwitch

CONFIGURING VXLAN AND GRE TUNNELS ON OPENVSWITCH

UPDATE: VXLAN ENCAPSULATION

VXLan is now upstreamed into the Master build. It is worth mentioning that is the encapsulation only, not the Multicast control plane functionality. That is being worked on by Mestery currently. So basically no need to pull from the fork but pull directly from the Open vSwitch Master. ‘git clone git://openvswitch.org/openvswitch’ have the VXLan encap included.

I have done a couple of GRE tunnel how-tos using OpenvSwitch (OVS). I ran across the git tree of the VXlan implementation which is still in development I am sure. I don’t want anyone to think this is anything other than me seeing the tree on GitHub. I had been itching to give VXLan a spin in OVS so why not ferret out someones tree on GitHub. I believe VXLan is still scheduled to officially release soon in OpenvSwitch. So here are the steps for installing, configuring tunnels on OpenvSwitch with VXLan and GRE encapsulations. At the end we will compare some of the protocols with difference MTU sizes. The results were interesting I think (for a nerd). We will be installing and then configure both GRE and VXLan encapsulated tunnels using Open vSwitch.

I like seeing some collaboration of really smart people from different companies as displayed in the GPL below.

/* * Copyright (c) 2011 Nicira Networks.
* Copyright (c) 2012 Cisco Systems Inc.
* Distributed under the terms of the GNU GPL version 2.
* * Significant portions of this file may be copied from parts of the Linux
* kernel, by Linus Torvalds and others. */

Figure 1. Example of how tunnels can be leveraged as overlays.

By the way I should probably disclaim now that huge Layer2 networks do not scale and huge Layer3 networks do. Host count in a broadcast domain/Vlan/network should be kept to a reasonable 3 digit number. Cisco is quick to point out that OTV is the solution over WAN’s for extending Layer 2 networks with OTV as the solution to extend Layer 2 Vlans. That said Overlays are flat out required to overcome Vlan number limitations and have lots of potential with programmatic orchestration.

Figure 2. OVS punts the first packet to user land for the forwarding decision and passes the data path back to the data path in the kernel for subsequent packets in the flow. slow path for the first packet then fast path for the rest.

I do some simple Iperf tests at the end of the post with 1500 and 9000 byte MTUs. The numbers are kind of fun to look at from the result. For a really nice analysis take a look at Martin’s post at Network Heresy on comparing STT, Linux Bridging and GRE. I am going to spin up some VMs on the VXLan tunnel later this week and measure the speeds a little closer and see how GRE and VXLan stack up to one another from hosts using overlays. I just ran some Iperfs from the hypervisor itself rather than VMs. OVS also supports Capwap encapsulation that performs mac in GRE which is rather slick. Wonder why we do not hear much about that. I am going to dig in when I get back to home from the road later next week.

Figure 3. Lab Setup basically has a fake interface up with Br1. Real world Br1 would have VMs tapped on it. The video uses br1 and br2 but it got to be confusing for people so I changed it to br0 and br1 to match most peoples eth0 = br0 NIC naming in ifconfig.

Quick Video of the fast install below.

---

For those familiar with the build you can just paste the following in your bash shell as root. To walk through the install skip the following snippet.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

apt - get update apt - get install - y git python - simplejson python - qt4 python - twisted - conch automake autoconf gcc uml - utilities libtool build - essential git pkg - config linux - headers - ` uname - r ` git clone git : //openvswitch.org/openvswitch cd openvswitch . / boot . sh . / configure -- with - linux = / lib / modules / ` uname - r ` / build make & amp ; & amp ; make install insmod datapath / linux / openvswitch . ko touch / usr / local / etc / ovs - vswitchd . conf mkdir - p / usr / local / etc / openvswitch ovsdb - tool create / usr / local / etc / openvswitch / conf . db vswitchd / vswitch . ovsschema ovsdb - server / usr / local / etc / openvswitch / conf . db \ -- remote = punix : / usr / local / var / run / openvswitch / db . sock \ -- remote = db : Open_vSwitch , manager_options \ -- private - key = db : SSL , private_key \ -- certificate = db : SSL , certificate \ -- bootstrap - ca - cert = db : SSL , ca_cert -- pidfile -- detach -- log - file ovs - vsctl -- no - wait init ovs - vswitchd -- pidfile -- detach ovs - vsctl show

OPEN VSWITCH SYSTEM PREPERATION

This is on two boxes with 64 cores and Broadcom Corporation NetXtreme II BCM57810 10G Nics to a random 10G TOR switch.
Install dependencies

1 2	; html - script : false ] apt - get update apt - get install python - simplejson python - qt4 python - twisted - conch automake autoconf gcc uml - utilities libtool build - essential git pkg - config

Download the  Open vSwitch latest build

1 2	git clone git : //openvswitch.org/openvswitch cd openvswitch

COMPILE OPENVSWITCH FROM SOURCE

1 2 3 4 5 6

; html - script : false ] . / boot . sh . / configure -- with - linux = / lib / modules / ` uname - r ` / build make make install #Load the OVS Kernel Module insmod datapath / linux / openvswitch . ko

1 2 3 4

; html - script : false ] #If you are running a stripped version of Nix like an EC2 cloud image and get an error along the lines of this: configure : error : source dir / lib / modules / 3.2.0 - 23 - virtual / build doesn & #039;t exist #Pull down the headers for your kernel. sudo apt - get install linux - headers - ` uname - r `

INITIAL OPEN VSWITCH CONFIGURATION

1 2 3

; html - script : false ] touch / usr / local / etc / ovs - vswitchd . conf mkdir - p / usr / local / etc / openvswitch ovsdb - tool create / usr / local / etc / openvswitch / conf . db vswitchd / vswitch . ovsschema

Run the following commands, note some dashes are two ‘- -’ e.g. – -remote=db & – -private-key

Start ovsdb-server, this stores the config into a file that is persistent even after restarts.

1 2 3 4 5 6

; html - script : false ] ovsdb - server / usr / local / etc / openvswitch / conf . db \ -- remote = punix : / usr / local / var / run / openvswitch / db . sock \ -- remote = db : Open_vSwitch , manager_options \ -- private - key = db : SSL , private_key \ -- certificate = db : SSL , certificate \ -- bootstrap - ca - cert = db : SSL , ca_cert -- pidfile -- detach -- log - file

1 2 3 4 5 6 7 8

; html - script : false ] #Only need to run this the first time. ovs - vsctl -- no - wait init #Start vswitch ovs - vswitchd -- pidfile -- detach #Verify the kernel module(s) in case you didn't earlier and get errors. #lsmod | grep br #brcompat               13512  0 #openvswitch            98196  1 brcompat

*note* “brcompat” is depreciated since the OVS upstream. Output should just be “openvswitch” as a loaded kernel module. If they are not there try loading again and check your path to the kernel module.

1	insmod datapath / linux / openvswitch . ko

At this point you have a fucntioning vanilla OVS install. Output should look something like this.

1 2 3 4 5

; html - script : false ] ovs - vsctl show b6d574d7 - 5582 - 4cc0 - 93e5 - a90cf0eb0a38 root @ demo - 139 : / home / ubuntu / ovs - vxlan # ovs-vsctl --version ovs - vsctl ( Open vSwitch ) 1.8.90 Compiled Aug 19 2012 06 : 23 : 36

CONFIGURE LINUX NETWORKING

I have one NIC (eth0) on the same LAN segment/network/vlan.
We are attaching eth0 to br1 and applying an IP to the bridge interface.
We are attaching an IP to br1. br1 is the island that we are building a tunnel for hosts to connect on. Without the VXLAN tunnel, the two br1 interfaces should not be able to ping one another. Note: This is being setup on the same subnet, it is important to keep in mind that the VXLAN framing will allow for the tunnel to be established over disparate networks. E.g. can be done over the Internet etc.

---

Configuration for Host 1

1 2 3 4 5 6 7 8

; html - script : false ] #Host 1 Configuration ovs - vsctl add - br br0 ovs - vsctl add - br br1 ovs - vsctl add - port br0 eth0 ifconfig eth0 0 & amp ; & amp ; ifconfig br0 192.168.1.10 netmask 255.255.255.0 route add default gw 192.168.1.1 br0 ifconfig br1 10.1.2.10 netmask 255.255.255.0 ovs - vsctl add - port br1 gre1 -- set interface gre1 type = gre options : remote_ip = 192.168.1.11

---

Configuration for Host 2

1 2 3 4 5 6 7 8

; html - script : false ] #Host 1 Configuration ovs - vsctl add - br br0 ovs - vsctl add - br br1 ovs - vsctl add - port br0 eth0 ifconfig eth0 0 & amp ; & amp ; ifconfig br0 192.168.1.11 netmask 255.255.255.0 route add default gw 192.168.1.1 br0 ifconfig br1 10.1.2.11 netmask 255.255.255.0 ovs - vsctl add - port br1 gre1 -- set interface gre1 type = gre options : remote_ip = 192.168.1.10

If you have issues getting the bridge built you may need to kill the OVS processes and restart them depending on your step order.

Your Linux routing table and ifconfig should now look something like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

root @ ub64 : / home / brent / openvswitch # route -n Kernel IP routing table Destination     Gateway         Genmask         Flags Metric Ref     Use Iface 0.0.0.0          192.168.1.1      0.0.0.0          UG      0        0          0 br0 10.1.2.0          0.0.0.0          255.255.255.0    U      0        0          0 br1 192.168.1.0      0.0.0.0          255.255.255.0    U      0        0          0 br0 root @ ub64 : / home / brent / openvswitch # ifconfig br0       Link encap : Ethernet   HWaddr 00 : 0c : 29 : 65 : fd : 82               inet addr : 192.168.1.10    Bcast : 192.168.1.255    Mask : 255.255.255.0            inet6 addr : fe80 :: 285f : 3ff : fea2 : dafc / 64 Scope : Link            UP BROADCAST RUNNING   MTU : 1500    Metric : 1            RX packets : 622 errors : 0 dropped : 0 overruns : 0 frame : 0            TX packets : 54 errors : 0 dropped : 0 overruns : 0 carrier : 0            collisions : 0 txqueuelen : 0            RX bytes : 171756 ( 171.7 KB )    TX bytes : 3832 ( 3.8 KB )   br1       Link encap : Ethernet   HWaddr ea : 34 : 7d : 49 : e7 : 49               inet addr : 10.1.2.10    Bcast : 10.1.2.255    Mask : 255.255.255.0            inet6 addr : fe80 :: c0a7 : aaff : fe90 : 29dc / 64 Scope : Link            UP BROADCAST RUNNING   MTU : 1500    Metric : 1            RX packets : 12 errors : 0 dropped : 0 overruns : 0 frame : 0            TX packets : 13 errors : 0 dropped : 0 overruns : 0 carrier : 0            collisions : 0 txqueuelen : 0            RX bytes : 896 ( 896.0 B )    TX bytes : 986 ( 986.0 B )   eth0       Link encap : Ethernet   HWaddr 00 : 0c : 29 : 65 : fd : 82               inet6 addr : fe80 :: 20c : 29ff : fe65 : fd82 / 64 Scope : Link            UP BROADCAST RUNNING MULTICAST   MTU : 1500    Metric : 1            RX packets : 1222493 errors : 0 dropped : 0 overruns : 0 frame : 0            TX packets : 53824 errors : 0 dropped : 0 overruns : 0 carrier : 0            collisions : 0 txqueuelen : 1000            RX bytes : 653399936 ( 653.3 MB )    TX bytes : 4089412 ( 4.0 MB )   lo         Link encap : Local Loopback              inet addr : 127.0.0.1    Mask : 255.0.0.0            inet6 addr : :: 1 / 128 Scope : Host            UP LOOPBACK RUNNING   MTU : 16436    Metric : 1            RX packets : 7088 errors : 0 dropped : 0 overruns : 0 frame : 0            TX packets : 7088 errors : 0 dropped : 0 overruns : 0 carrier : 0            collisions : 0 txqueuelen : 0            RX bytes : 537609 ( 537.6 KB )    TX bytes : 537609 ( 537.6 KB )

TROUBLESHOOTING OPEN VSWITCH INSTALLATION

An error was sent to me by someone that looked like this below, restarting the OVS procs will clear it up:

1 2 3 4 5 6 7 8 9 10

; html - script : false ] Aug 19 06 : 34 : 04 demo - 139 ovs - vsctl : 00001 | vsctl | INFO | Called as ovs - vsctl del - br br1 Aug 19 06 : 34 : 04 demo - 139 ovs - vswitchd : 00026 | dpif | WARN | failed to enumerate system datapaths : No such file or directory Aug 19 06 : 34 : 04 demo - 139 ovs - vswitchd : 00027 | bridge | INFO | ovs - vswitchd ( Open vSwitch ) 1.8.90 Aug 19 06 : 34 : 20 demo - 139 ovs - vsctl : 00001 | vsctl | INFO | Called as ovs - vsctl add - br br1 Aug 19 06 : 34 : 20 demo - 139 ovs - vswitchd : 00028 | dpif | WARN | failed to enumerate system datapaths : No such file or directory Aug 19 06 : 34 : 20 demo - 139 ovs - vswitchd : 00029 | dpif | WARN | failed to create datapath br1 : No such file or directory Aug 19 06 : 34 : 20 demo - 139 ovs - vswitchd : 00030 | ofproto_dpif | ERR | failed to open datapath br1 : No such file or directory Aug 19 06 : 34 : 20 demo - 139 ovs - vswitchd : 00031 | ofproto | ERR | failed to open datapath br1 : No such file or directory Aug 19 06 : 34 : 20 demo - 139 ovs - vswitchd : 00032 | bridge | ERR | failed to create bridge br1 : No such file or directory Aug 19 06 : 34 : 20 demo - 139 ovs - vswitchd : 00033 | bridge | INFO | ovs - vswitchd ( Open vSwitch ) 1.8.90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

; html - script : false ] #Find the OVS PIDs from PS ps - ea | grep ovs 16369 ?          00 : 00 : 00 ovsdb - server 16372 ?          00 : 00 : 00 ovs - vswitchd 16373 ?          00 : 00 : 00 ovs - vswitchd 16375 ?          00 : 00 : 00 ovs_workq #Terminate them kill 16369 kill 16372 kill 16375 #Then restart the OVS server ovsdb - server / usr / local / etc / openvswitch / conf . db \ -- remote = punix : / usr / local / var / run / openvswitch / db . sock \ -- remote = db : Open_vSwitch , manager_options \ -- private - key = db : SSL , private_key \ -- certificate = db : SSL , certificate \ -- bootstrap - ca - cert = db : SSL , ca_cert -- pidfile -- detach -- log - file #Start vswitchd (JSON DB) ovs - vswitchd -- pidfile -- detach

BUILD A GRE TUNNEL IN OPEN VSWITCH

1 2	; html - script : false ] ovs - vsctl add - port br1 gre1 -- set interface gre1 type = gre options : remote_ip = 192.168.1.10 ovs - vsctl add - port br1 gre1 -- set interface gre1 type = gre options : remote_ip = 192.168.1.11

Adjust MTUs with % ifconfig <interface e.g. br0 eth0 etc> mtu 9000
#(Remember your switch needs to have >= your MTU.

Figure 4. Output of GRE Tunnels running with 1500 byte MTUs.

Figure 5. Output of GRE Tunnels running with 9000 byte MTUs.

---

Now delete the GRE ports

1	ovs - vsctl del - port gre1

CONFIGURE THE VXLAN TUNNEL

The difference here is the “type” specifying the tunnel encapsulation.

1 2	ovs - vsctl add - port br1 vx1 -- set interface vx1 type = vxlan options : remote_ip = 192.168.1.10 ovs - vsctl add - port br1 vx1 -- set interface vx1 type = vxlan options : remote_ip = 192.168.1.11

Figure 6 Output of VXLan Tunnels running with 1500 byte MTUs.