Vitual Router in The Cloud

VyOS and ESXi,VyOS Configuration

The next step is to configure both VyOS routers. Before we do, we should ensure that we have a good high-level understanding of what should be happening.

The ultimate goal of this three-router setup is to have our own VyOS router as the gateway to the Internet, while also allowing the Verizon router to continue providing network access for the value-added services like:

• Video-on-Demand to set-top boxes
• On-screen caller ID
• Remote DVR access
• Etc.

The Verizon router does this by setting up its own NAT’d network on the 192.168.1.0/24 range, which the STBs in the house sit on and use to communicate with Verizon’s servers. The VZ router expects and requires the IP it is assigned on its WAN port to be publicly routable on the FiOS ISP network. If it is not, things may or may not work, or they might become unpredictable in their functionality.

The entire point of the secondary router is to provide 1:1 NAT between the home network and the VZ router, so that the VZ router gets assigned the same IP as the primary router that is actually talking to the FiOS ISP network.

With three different Layer 2 domains and some creative port forwarding, the Verizon router won’t even know the difference.

This network configuration, combined with some port forwarding rules on the primary and secondary router (discussed later), allows traffic between the Verizon router and the Verizon servers to flow normally without the VZ router being aware that it is not actually directly connected to the FiOS ISP network.

Let’s start by configuring the primary router. This router will actually receive the public-facing IP from the FiOS ISP network, and thus will ultimately be responsible for all Internet traffic. Log into your primary router and run the show interfaces command.

vyos@primary-router:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             108.0.0.123/24                    u/u  FiOS Public Internet 
eth1             10.0.0.1/24                       u/u  Home Network 
lo               127.0.0.1/8                       u/u  
                 ::1/128

We see two Ethernet interfaces, eth0 and eth1. These represent the two vNICs provisioned to this VM, and which correspond to the FiOS Public Network and Home Network port groups, respectively.

Let’s configure the eth0 interface first.

vyos@primary-router:~$ configure
vyos@primary-router:~# set interfaces ethernet eth0 address dhcp
vyos@primary-router:~# set interfaces ethernet eth0 description FiOS_ISP_Net
vyos@primary-router:~# set interfaces ethernet eth0 duplex auto
vyos@primary-router:~# set interfaces ethernet eth0 speed auto

This will set this interface up to use a dynamically assigned address (from Verizon), set a description to make it easy to remember what it connects to, and auto negotiate speed and duplex settings.

There is one more step required. We must configure this interface to impersonate our Verizon hardware router’s WAN interface by setting it to use the same MAC address (Verizon filters MACs that are not on its whitelist). You can find the WAN MAC you need to enter printed on the bottom of your Verizon router. Replace 0a:1b:2c:3d:4e:5f below as appropriate:

vyos@primary-router:~# set interfaces ethernet eth0 mac 0a:1b:2c:3d:4e:5f

Let’s take a look at the changes we are making.

vyos@primary-router:~# compare

When you are satisfied, commit the changes to the running configuration and save the running config to disk. If you commit but do not save, the changes will not persist after a reboot of the router.

vyos@primary-router:~# commit
vyos@primary-router:~# save

Let’s take another look at the interface configuration now. We’re still in configuration mode (note the # symbol at the end of the command prompt), so we need to prepend run to the command we used before.

vyos@primary-router:~# run show interfaces

Hopefully, your eth0 interface has a public address assigned from the Verizon DHCP server. If not, check your connections and configurations.

Assuming all is well, you should now be able to ping addresses to confirm that you have connectivity out to the Internet.

vyos@primary-router:~# run ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=251 time=21.1 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=251 time=22.0 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=251 time=20.9 ms
64 bytes from 8.8.8.8: icmp_req=4 ttl=251 time=22.3 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 20.948/21.610/22.307/0.605 ms

If this works, then congratulations! The good news is that your first router is working. The bad news is that nothing else can use your Internet connection yet.

VyOS and OpenStack Configuration Drives

VyOS is an open source fork of the last open source release of Vayatta, which turned proprietary a few years ago. We are currently using VyOS at work to set up OSPF routers in an OpenStack environment, and will soon have to spawn a very large amount of these in a proof-of-concept deployment.

This describes how we add support for OpenStack’s Configuration Drive to VyOS.

VyOS has something of an unhealthy relationship with Debian Squeeze (it is currently incompatible with newer Debian releases), and requires a Debian Squeeze installation in order to create the VyOS ISO used for deployments.

Below we will patch a post-installation script and add our own (very simple) Python-script that parses the Configuration Drive information and complies with a very small subset of what features packages like cloud-init provide. Unfortunately, cloud-init is not available for Debian Squeeze, which is the whole reason we are doing this in the first place.

Steps:

1. Install Squeeze
2. Create Python script
3. Run setup-script
4. Import ISO into Glance
5. Spawn OpenStack instance
6. Verify that it works

Prepare a little Python script for parsing the OpenStack Configuration Drive metadata:

#!/usr/bin/python
import json
import shutil

meta_data_file = open('/config-drive/openstack/latest/meta_data.json')
json_input = meta_data_file.read()

try:
  decoded = json.loads(json_input)

  for file in decoded['files']:
    print file['content_path'], file['path']
    shutil.copy2('/config-drive/openstack' + file['content_path'], file['path'])

except (ValueError, KeyError, TypeError):
  print "JSON format or content error"

Save this as process-openstack-metadata.py. This will be baked into the ISO in the script below as/root/vyos-init.py.

Below is a script to generate a VyOS ISO with a few modifications. Most of it is straight from the VyOS wiki page How to build an ISO image. Read through it so you see what it does and save it as build-vyos-iso.sh, chmod it (chmod +x build-vyos-iso.sh) and run it.

#!/bin/bash -xe

apt-get install debian-archive-keyring

cat >> /etc/apt/sources.list <<EOF
deb http://backports.debian.org/debian-backports squeeze-backports main
EOF

apt-get update

# Get backported version of squashfs
apt-get -t squeeze-backports install squashfs-tools

apt-get install git autoconf automake dpkg-dev live-helper syslinux genisoimage

branch=hydrogen # hydrogen = stable, helium = dev

if ! test -d build-iso
then
  git clone https://github.com/vyos/build-iso.git

  cd build-iso

  git branch $branch --track origin/$branch
  git checkout $branch
else
  cd build-iso
fi

if ! test -d pkgs/vyatta-cfg-system/debian
then
  git submodule update --init pkgs/vyatta-cfg-system

  cd pkgs/vyatta-cfg-system/

  git branch $branch --track origin/$branch
  git checkout $branch
else
  cd pkgs/vyatta-cfg-system/
fi

# Reset debian/vyatta-cfg-system.postinst.in so we can patch it again
git checkout debian/vyatta-cfg-system.postinst.in

# Patch debian/vyatta-cfg-system.postinst.in
patch -p0 <<"HEREDOC"
--- debian/vyatta-cfg-system.postinst.in    2015-01-17 15:09:53.000000000 +0100
+++ debian/vyatta-cfg-system.postinst.in.patched    2015-01-17 15:11:19.000000000 +0100
@@ -143,6 +143,19 @@
 # configuration is fully applied. Any modifications done to work around
 # unfixed bugs and implement enhancements which are not complete in the Vyatta
 # system can be placed here.
+
+mkdir /config-drive
+
+mount -o ro -t iso9660 /dev/disk/by-label/config-2 /config-drive
+
+/root/vyos-init.py
+
+configure
+load /root/configuration
+commit
+save
+
+umount /config-drive
 EOF
 fi

HEREDOC

cd -

mkdir livecd/config.vyatta/chroot_local-includes/root

cp ../process-openstack-metadata.py \
  livecd/config.vyatta/chroot_local-includes/root/vyos-init.py

chmod +x livecd/config.vyatta/chroot_local-includes/root/vyos-init.py

aptitude install pdebuild-cross
make vyatta-cfg-system

find pkgs -name 'vyatta-cfg-system*.deb' || \
  (echo "pkgs/vyatta-cfg-system*.deb not found, exiting..."; exit 1)

echo python-simplejson >> \
  livecd/config.vyatta/chroot_local-packageslists/vyatta-extra.list

export PATH=/sbin:/usr/sbin:$PATH
autoreconf -i
./configure
make iso

ls -l livecd/binary.iso

echo Done!

If everything went well you will have an ISO at ./build-iso/livecd/binary.iso.

Upload this file into OpenStack with Glance and name it “VyOS Router”:

glance image-create --name "VyOS Router" --is-public True \
  --disk-format iso --container bare < ./build-iso/livecd/binary.iso

Create your own config.boot (or whatever else you want on the deployed machine):

cat > config.boot <<"EOF"
interfaces {
    ethernet eth0 {
        address dhcp
    }
    loopback lo {
    }
}
service {
    ssh {
        port 22
    }
}
system {
    login {
        user vyos {
            authentication {
                plaintext-password "demo"
            }
            level admin
        }
    }
}
EOF

Spawn an instance with a predefined flavor and our new configuration file to be included on the configuration drive:

nova boot --config-drive true --image "VyOS Router" \
  --flavor <flavor> --file /root/configuration=config.boot \
  --meta essential=false --nic net-id=<net-id> vyos

Verify that it works by logging into VyOS and check if the running configuration is the one you expect. You can start by checking if the file /root/configuration exists and if it the content is what you intended.

How to debug:

vyos@vyos:~$ /usr/sbin/tcpdump -f "icmp" -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:10:13.234909 IP 10.168.11.102 > 8.8.8.8: ICMP echo request, id 40962, seq 49, length 64
18:10:13.261277 IP 8.8.8.8 > 10.168.11.102: ICMP echo reply, id 40962, seq 49, length 64
18:10:14.235045 IP 10.168.11.102 > 8.8.8.8: ICMP echo request, id 40962, seq 50, length 64
18:10:14.261379 IP 8.8.8.8 > 10.168.11.102: ICMP echo reply, id 40962, seq 50, length 64
18:10:15.235249 IP 10.168.11.102 > 8.8.8.8: ICMP echo request, id 40962, seq 51, length 64
18:10:15.261549 IP 8.8.8.8 > 10.168.11.102: ICMP echo reply, id 40962, seq 51, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
vyos@vyos:~$ /usr/sbin/tcpdump -f "icmp" -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
18:10:18.235887 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 54, length 64
18:10:18.262249 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 54, length 64
18:10:19.236110 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 55, length 64
18:10:19.262477 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 55, length 64
18:10:20.236345 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 56, length 64
18:10:20.262652 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 56, length 64
18:10:21.236527 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 57, length 64
18:10:21.262927 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 57, length 64
18:10:22.237082 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 58, length 64
18:10:22.263398 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 58, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
vyos@vyos:~$ ping 10.168.11.102
PING 10.168.11.102 (10.168.11.102) 56(84) bytes of data.
64 bytes from 10.168.11.102: icmp_req=1 ttl=64 time=0.481 ms
64 bytes from 10.168.11.102: icmp_req=2 ttl=64 time=0.559 ms