Inter-VRF Routing with VRF Lite

In Intro to VRF lite, we looked at how virtual routing and forwarding (VRF) instances can be employed to logically separate the layer three topologies of unrelated entities sharing a single physical infrastructure. VRFs work at layer three much like VLANs work at layer two, by assigning interfaces to a virtual domain isolated from other virtual domains at the same layer. This is a simple concept to grasp so long as each of the VRFs are maintained in isolation. However, things get a bit more complex when you need to route traffic from one VRF to another.

Consider the following topology, which might be found at a small business park or college campus.

The network pictured serves three customers: we'll call them customers Red, Green, and Blue. Each customer has its own Internet connection, housed in building one. Customer Green has employees in building two, customer Blue has employees in building three, and customer Red has employees in both buildings two and three. Additionally, VOIP service and connection to the PSTN (represented for now by a loopback interface) will be provided in the future by the network operator to customers Red and Green, but not Blue.

Buildings two and three are connected to building one via 802.1Q trunk links. All three customers rely on the same physical network infrastructure for connectivity, but the traffic from each customer must be isolated from the others for security reasons. The IP prefixes in use are as follows:

• Red: 172.16.0.0/16
• Green: 172.17.0.0/16
• Blue: 172.18.0.0/16
• VOIP Services: 192.168.99.0/24

The design question posed by this scenario is as follows: how can we provide local VOIP connectivity to some customers while keeping all of the customers isolated from one another?

Modifying the Multilayer Switches' SDM Template

Before we can implement VRFs on our multilayer switches (Catalyst 3550s, in this case), we have to modify how the internal ternary content-addressable memory (TCAM) is partitioned to store IP routes. Observe what happens if we attempt to enable VRF with the default SDM template:

S1(config)# ip routing
S1(config)# ip vrf Red
S1(config-vrf)#
%L3TCAM-3-SIZE_CONFLICT: VRF requires enabling extended routing

VRF routes require that an additional unique identifier, a route distinguisher, is prepended to each VRF route in the routing table (we'll cover this in more detail shortly). To accommodate for this, the TCAM must be repartitioned. (This step is only necessary when configuring VRFs on certain multilayer switches.)

S1(config)# sdm prefer extended-match
Changes to the running SDM preferences have been stored, but cannot take effect 
until the next reload.
Use 'show sdm prefer' to see what SDM preference is currently active.
S1(config)# ^Z
S1# show sdm prefer
 The current template is the default template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1K VLANs.

number of unicast mac addresses:   5K
 number of igmp groups:             1K
 number of qos aces:                1K
 number of security aces:           1K
 number of unicast routes:          8K
 number of multicast routes:        1K

The template stored for use after the next reload
 is the default extended-match template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1K VLANs.

number of unicast mac addresses:   5K
 number of igmp groups:             1K
 number of qos aces:                1K
 number of security aces:           1K
 number of unicast routes:          4K
 number of multicast routes:        1K

Note that the supported number of unicast routes has been halved from 8K to 4K.

We need to apply this change to and reload all three switches before proceeding.

Creating VRFs

We need to create four VRFs on S1: one for each of the three customers, and one for the shared VOIP network. Each VRF must be assigned a unique route distinguisher. A route distinguisher is simply a unique number stored beside each route in the routing table to associate it with its VRF and maintain uniqueness should two or more VRFs have overlapping address space (for example, if all three of our customers were addressed out of the 10.0.0.0/8 network).

There are two formats in which we can define a route distinguisher: <ASN>:<number> or <IP address>:<number>, where number is an arbitrary decimal number. Which method we choose is arbitrary. While the format of route distinguishers is an important design consideration in "real" VPNs, in VRF lite (i.e. VRFs without MPLS) it is only a locally-significant value. We'll use the <ASN>:<number> format with the private ASN 65000 (which will also be used for our BGP configuration later in the article) and number customers sequentially beginning from 1.

S1

ip routing
!
ip vrf Red
 rd 65000:1
!
ip vrf Green
 rd 65000:2
!
ip vrf Blue
 rd 65000:3
!
ip vrf Shared
 rd 65000:99

Next, we assign the physical and logical interfaces to VRFs and address them appropriately. We're using a loopback interface in place of a physical interface on S1 to emulate the shared VOIP network, however there is no effective difference with regard to VRF configuration. Note that interfaces must be assigned to a VRF before being addressed; assigning an interface to a VRF wipes any IP addresses already configured on that interface.

S1

vlan 16
vlan 17
vlan 18
!
interface Loopback99
 description VOIP Services
 ip vrf forwarding Shared
 ip address 192.168.99.1 255.255.255.0
!
interface FastEthernet0/1
 no switchport
 ip vrf forwarding Red
 ip address 172.16.1.2 255.255.255.252
!
interface FastEthernet0/3
 no switchport
 ip vrf forwarding Green
 ip address 172.17.1.2 255.255.255.252
!
interface FastEthernet0/5
 no switchport
 ip vrf forwarding Blue
 ip address 172.18.1.2 255.255.255.252
!
interface FastEthernet0/13
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface FastEthernet0/16
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan16
 ip vrf forwarding Red
 ip address 172.16.0.1 255.255.255.0
!
interface Vlan17
 ip vrf forwarding Green
 ip address 172.17.0.1 255.255.255.0
!
interface Vlan18
 ip vrf forwarding Blue
 ip address 172.18.0.1 255.255.255.0

The physical trunk interfaces F0/13 and F0/16 are not assigned to a VRF, because they are not routed interfaces. VRF configuration is only relevant at layer three.

We can use show ip vrf to examine the VRFs and their assigned interfaces:

S1# show ip vrf
  Name                             Default RD          Interfaces
  Blue                             65000:3             Fa0/5                  
                                                       Vl18
  Green                            65000:2             Fa0/3                  
                                                       Vl17
  Red                              65000:1             Fa0/1                  
                                                       Vl16
  Shared                           65000:99            Lo99                   
S1# show ip vrf interfaces
Interface              IP-Address      VRF                              Protocol
Fa0/5                  172.18.1.2      Blue                             up      
Vl18                   172.18.0.1      Blue                             up      
Fa0/3                  172.17.1.2      Green                            up      
Vl17                   172.17.0.1      Green                            up      
Fa0/1                  172.16.1.2      Red                              up      
Vl16                   172.16.0.1      Red                              up      
Lo99                   192.168.99.1    Shared                           up      

We only need to create two VRFs on each of the two other switches, as each connects only two customers, and assign the appropriate interfaces.

S2

ip routing
!
ip vrf Red
 rd 65000:1
!
ip vrf Green
 rd 65000:2
!
vlan 16
vlan 17
vlan 216
vlan 217
!
interface FastEthernet0/13
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan16
 ip vrf forwarding Red
 ip address 172.16.0.2 255.255.255.0
!
interface Vlan17
 ip vrf forwarding Green
 ip address 172.17.0.2 255.255.255.0
!
interface Vlan216
 ip vrf forwarding Red
 ip address 172.16.2.1 255.255.255.0
!
interface Vlan217
 ip vrf forwarding Green
 ip address 172.17.2.1 255.255.255.0

S3

ip routing
!
ip vrf Red
 rd 65000:1
!
ip vrf Blue
 rd 65000:3
!
vlan 16
vlan 18
vlan 316
vlan 318
!
interface FastEthernet0/13
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan16
 ip vrf forwarding Red
 ip address 172.16.0.3 255.255.255.0
!
interface Vlan18
 ip vrf forwarding Blue
 ip address 172.18.0.3 255.255.255.0
!
interface Vlan316
 ip vrf forwarding Red
 ip address 172.16.3.1 255.255.255.0
!
interface Vlan318
 ip vrf forwarding Blue
 ip address 172.18.3.1 255.255.255.0

Configuring OSPF

Next we'll configure OSPF to exchange routes among the three buildings. Three independent OSPF processes are to be run, one per customer VRF. OSPF configuration here is pretty straight forward, as we can simply place all interfaces in area 0 within each VRF.

S1

router ospf 1 vrf Red
 network 0.0.0.0 255.255.255.255 area 0
!
router ospf 2 vrf Green
 network 0.0.0.0 255.255.255.255 area 0
!
router ospf 3 vrf Blue
 network 0.0.0.0 255.255.255.255 area 0

S2

router ospf 1 vrf Red
 network 0.0.0.0 255.255.255.255 area 0
 passive-interface Vlan216
!
router ospf 2 vrf Green
 network 0.0.0.0 255.255.255.255 area 0
 passive-interface Vlan217
!

S3

router ospf 1 vrf Red
 network 0.0.0.0 255.255.255.255 area 0
 passive-interface Vlan316
!
router ospf 3 vrf Blue
 network 0.0.0.0 255.255.255.255 area 0
 passive-interface Vlan318

We can verify that OSPF adjacencies have been appropriately established with show ip ospf neighbor. Note that this command is not VRF-specific.

S1# show ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
172.18.3.1        1   FULL/DR         00:00:31    172.18.0.3      Vlan18
172.17.2.1        1   FULL/DR         00:00:32    172.17.0.2      Vlan17
172.16.2.1        1   FULL/BDR        00:00:32    172.16.0.2      Vlan16
172.16.3.1        1   FULL/DR         00:00:31    172.16.0.3      Vlan16

At this point, all customers in all buildings should have full connectivity within their respective VRFs.

S2# show ip route vrf Red

Routing Table: Red
...

172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C       172.16.0.0/24 is directly connected, Vlan16
O       172.16.1.0/30 [110/2] via 172.16.0.1, 00:00:31, Vlan16
C       172.16.2.0/24 is directly connected, Vlan216
O       172.16.3.0/24 [110/2] via 172.16.0.3, 00:00:31, Vlan16
S2# show ip route vrf Green

Routing Table: Green
...

172.17.0.0/16 is variably subnetted, 3 subnets, 2 masks
O       172.17.1.0/30 [110/2] via 172.17.0.1, 00:00:37, Vlan17
C       172.17.0.0/24 is directly connected, Vlan17
C       172.17.2.0/24 is directly connected, Vlan217

S3# show ip route vrf Red

Routing Table: Red
...

172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C       172.16.0.0/24 is directly connected, Vlan16
O       172.16.1.0/30 [110/2] via 172.16.0.1, 00:01:04, Vlan16
O       172.16.2.0/24 [110/2] via 172.16.0.2, 00:01:04, Vlan16
C       172.16.3.0/24 is directly connected, Vlan316
S3# show ip route vrf Blue

Routing Table: Blue
...

172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks
C       172.18.3.0/24 is directly connected, Vlan318
C       172.18.0.0/24 is directly connected, Vlan18
O       172.18.1.0/30 [110/2] via 172.18.0.1, 00:01:05, Vlan18

Configuring Multiprotocol BGP

Multiprotocol BGP (MP-BGP or BGP-MP) will be used only to exchange routes between VRFs, and therefore needs to be configured only on S1. We'll use the private AS number 65000 for the BGP process.

Because all of our routed interfaces on S1 have been assigned to VRFs, we'll create a loopback interface not in a VRF to keep BGP from complaining about not being able to find a router ID. (In the real world, one might consider using the IP of a management interface as the BGP router ID.) A router ID can also be configured per BGP address family.

S1(config)# interface loopback0
S1(config-if)# ip address 192.0.2.1 255.255.255.255

To configure MP-BGP, we configure a separate address family within BGP for each VRF and simply redistribute all connected and OSPF-learned routes within that VRF.

S1

router bgp 65000
 no synchronization
 bgp log-neighbor-changes
 no auto-summary
 !
 address-family ipv4 vrf Red
  redistribute connected
  redistribute ospf 1 vrf Red
  no synchronization
 exit-address-family
 !
 address-family ipv4 vrf Green
  redistribute connected
  redistribute ospf 2 vrf Green
  no synchronization
 exit-address-family
 !
 address-family ipv4 vrf Blue
  redistribute connected
  redistribute ospf 3 vrf Blue
  no synchronization
 exit-address-family
 !
 address-family ipv4 vrf Shared
  redistribute connected
  no synchronization
 exit-address-family

We can verify that each BGP address family now maintains routes for its respective VRF. (To view the BGP routes for only a specific VRF, use the command show ip bgp vpnv4 vrf.)

S1# show ip bgp vpnv4 all
BGP table version is 47, local router ID is 192.0.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65000:1 (default for vrf Red)
*> 172.16.0.0/24    0.0.0.0                  0         32768 ?
*> 172.16.1.0/30    0.0.0.0                  0         32768 ?
*> 172.16.2.0/24    172.16.0.2               2         32768 ?
*> 172.16.3.0/24    172.16.0.3               2         32768 ?
Route Distinguisher: 65000:2 (default for vrf Green)
*> 172.17.0.0/24    0.0.0.0                  0         32768 ?
*> 172.17.1.0/30    0.0.0.0                  0         32768 ?
*> 172.17.2.0/24    172.17.0.2               2         32768 ?
Route Distinguisher: 65000:3 (default for vrf Blue)
*> 172.18.0.0/24    0.0.0.0                  0         32768 ?
*> 172.18.1.0/30    0.0.0.0                  0         32768 ?
*> 172.18.3.0/24    172.18.0.3               2         32768 ?
Route Distinguisher: 65000:99 (default for vrf Shared)
*> 192.168.99.0     0.0.0.0                  0         32768 ?

Enabling VRF Route Import and Export

Our final task is to configure route import and export among the Red, Green, and Shared VRFs. VRF route import and export are configured as two explicit, independent functions. Under VRFs Red and Green, we need to:

• Export local routes
• Import routes from VRF Shared

Under VRF Shared:

• Export local routes
• Import routes from VRFs Red and Green

To do this, we configure each VRF with import and export route targets. A route target is similar in format to a route distinguisher, but serves a different purpose. Whereas any given VRF route in our scenario has exactly one route distinguisher to uniquely identify it, a route can have zero or more route targets as arbitrary identifiers for reference by import and export policies. This allows for much greater flexibility than if only the route distinguisher was used. Although used only locally in our example, route targets are communicated with MP-BGP neighbors as extended communities.

Route targets can be defined in the same formats as route distinguishers. Here each route target will reuse the same format and value of its VRF's route distinguisher, just to keep things simple. However, remember that there is no direct relation between the two. Additionally, although each VRF is exporting only one route-target in our example, multiple route-targets can be exported simultaneously (resulting in multiple communities attached to an MP-BGP route).

ip vrf Blue
 rd 65000:3
!
ip vrf Green
 rd 65000:2
 route-target export 65000:2
 route-target import 65000:99
!
ip vrf Red
 rd 65000:1
 route-target export 65000:1
 route-target import 65000:99
!
ip vrf Shared
 rd 65000:99
 route-target export 65000:99
 route-target import 65000:1
 route-target import 65000:2

Now VRF Shared has routes from VRFs Red and Green, and VRFs Red and Green each have the route from VRF Shared (it may take a minute for the routes to appear in the table):

S1# show ip route vrf Shared

Routing Table: Shared
...

172.17.0.0/16 is variably subnetted, 2 subnets, 2 masks
B       172.17.1.0/30 is directly connected, 00:00:24, FastEthernet0/3
B       172.17.0.0/24 is directly connected, 00:00:24, Vlan17
     172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
B       172.16.0.0/24 is directly connected, 00:00:24, Vlan16
B       172.16.1.0/30 is directly connected, 00:00:24, FastEthernet0/1
C    192.168.99.0/24 is directly connected, Loopback99
S1# show ip route vrf Red

Routing Table: Red
...

172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C       172.16.0.0/24 is directly connected, Vlan16
C       172.16.1.0/30 is directly connected, FastEthernet0/1
O       172.16.2.0/24 [110/2] via 172.16.0.2, 00:05:38, Vlan16
O       172.16.3.0/24 [110/2] via 172.16.0.3, 00:05:38, Vlan16
B    192.168.99.0/24 is directly connected, 00:00:44, Loopback99
S1# show ip route vrf Green

Routing Table: Green
...

172.17.0.0/16 is variably subnetted, 3 subnets, 2 masks
C       172.17.1.0/30 is directly connected, FastEthernet0/3
C       172.17.0.0/24 is directly connected, Vlan17
O       172.17.2.0/24 [110/2] via 172.17.0.2, 00:05:43, Vlan17
B    192.168.99.0/24 is directly connected, 00:00:49, Loopback99

VRF Blue, however, still contains only its own routes, as we did not configure import or export for this VRF:

S1# show ip route vrf Blue

Routing Table: Blue
...

172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks
O       172.18.3.0/24 [110/2] via 172.18.0.3, 00:06:02, Vlan18
C       172.18.0.0/24 is directly connected, Vlan18
C       172.18.1.0/30 is directly connected, FastEthernet0/5

Lastly, we need to configure the VRF OSPF processes to redistribute BGP-learned routes to OSPF peers on S1:

S1(config)# router ospf 1 vrf Red
S1(config-router)# redistribute bgp 65000 subnets
S1(config-router)# router ospf 2 vrf Green
S1(config-router)# redistribute bgp 65000 subnets

These routes are now propagated via OSPF to S2 and S3 within their respective VRFs. On S3, we can verify that VLAN 316 (VRF Red) has access to the VOIP network, but VLAN 318 (VRF Blue) does not:

S3# show ip route vrf Red

Routing Table: Red
...

172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C       172.16.0.0/24 is directly connected, Vlan16
O       172.16.1.0/30 [110/2] via 172.16.0.1, 00:10:15, Vlan16
O       172.16.2.0/24 [110/2] via 172.16.0.2, 00:10:15, Vlan16
C       172.16.3.0/24 is directly connected, Vlan316
O E2 192.168.99.0/24 [110/1] via 172.16.0.1, 00:10:15, Vlan16
S3# show ip route vrf Blue

Routing Table: Blue
...

172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks
C       172.18.3.0/24 is directly connected, Vlan318
C       172.18.0.0/24 is directly connected, Vlan18
O       172.18.1.0/30 [110/2] via 172.18.0.1, 00:25:17, Vlan18
S3# ping vrf Red 192.168.99.1 source vlan316

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.99.1, timeout is 2 seconds:
Packet sent with a source address of 172.16.3.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
S3# ping vrf Blue 192.168.99.1 source vlan318

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.99.1, timeout is 2 seconds:
Packet sent with a source address of 172.18.3.1 
.....
Success rate is 0 percent (0/5)

The final configurations of all three switches are made available here:

• S1.txt
• S2.txt
• S3.txt