This blog post is the first in a series covering Performance Routing (PfR) formerly known as Optimized Edge Routing (OER) that I will be publishing over the coming weeks. I decided to cover PfR in a series of blog posts contrary to a single post as PfR is a very powerful and useful feature that leverages the power of Cisco’s IOS but at the same time PfR is potentially very complicated and often confusing feature to configure and troubleshoot. Trying to cover PfR in a single blog post would be the equivalent of trying to cover OSPF in a single blog post. In fact if you compare the IOS 12.4T OSPF Configuration Guide against the Optimized Edge Routing (OER) Configuration Guide you will notice that OER documentation is nearly 35% larger.
NOTE
In this blog post the term PfR will be used in place of OER wherever possible as Cisco has started to depricate the OER terminology and commands as of IOS 15.0.
The primary focus of these blog posts will be how PfR relates to the Routing and Switching CCIE Lab Exam (PfR v2.2). The first couple posts will cover basic scenarios (static routing and BGP) with PfR, while introducing the reader to the PfR specific terminology and features as we use then and/or run across them. After we cover the basic scenarios I will get into more complicated scenarios using PfR to optimize routing based upon DSCP values, inbound routing optimization using BGP, routing based upon application response time and voice call quality. A final post will cover PfR in IOS 15.1 (PfR v3.0) and will focus on some of the newer PfR features. I will try to keep the details and complexities of PfR out of the first couple blog posts so that the reader can gain a solid grasp of PfR before moving forward. I spend roughly half a day in my new Routing and Switching CCIE 10 Day Bootcampcovering PfR as it’s important for the R&S CCIE candidate has a solid understanding before attempting the CCIE Lab exam. Additionally, I personally believe that in the future the concept of centralized route control and/or route manipulation as with PfR could become more common, similar to the concept of OpenFlow. With that being said lets get started.
Performance Routing (PfR), previously called Optimized Edge Routing (OER), introduces a new concept into IP routing. With traditional routing, path selection decisions do not consider a particular path’s traffic characteristics be it throughput, actual delay, packet loss, voice mean opinion score (MOS), monetary cost of a given path, or jitter. PfR enhances the classic destination-based routing concept centered on the shortest path (lowest-cost metric) by adding into the routing selection process, network performance and/or application performance aware intelligence. In the past when routing protocols were implemented in large-scale networks, routers did not have the resources to calculate the best path based upon anything other than a simple metric. Additionally, many of these networks would be considered simplistic in regard to the number of primary and redundant links compared to today’s networks. With the increase in CPU power and memory available in routers today, routing based solely upon a simple metric (hop count, cost, as-path length, etc.) is not the best use of these available resources. PfR will leverage these available resources to allow routing decisions based upon additional factors namely the networks performance and/or application performance across the network. Getting the most out of your network’s available bandwidth and/or the best possible performance for your applications across the network should be one of the primary goals of any network implementation. Let’s look at an example of how we can do this using PfR.
A site consisting of a single router has two connections to the Internet. One from service provider A and the other from service provider B. The link through provider A is a 10Mbps Ethernet link while the link through provider B is a 1.5Mbps T1 link. The Ethernet link is the primary path used to reach the Internet, and the T1 is used as a backup in the event the Ethernet link is down. Routing is done using static default route pointing out the Ethernet link and an additional static with a higher administrative distance (floating static) pointing out the T1 link. With this configuration, the T1 link is only used in the case the Ethernet link is down. This means the T1 link primarily sits idle for the vast majority of the time. The administrator could implement additional static routes pointing out the T1 link for networks that provider B originates to at least get some use out of the T1 but this is not an ideal solution. Ideally, the router itself will automatically select the best path for traffic based upon performance through the different service providers and/or allow traffic to overflow to the T1 link when the Ethernet link starts to become fully utilized. For simplify we will configure the latter case in this blog post.
In our scenario, the requirements are for the router to automatically move traffic flows to the T1 link when the Ethernet link becomes 75% utilized. Using traditional static routes, activating another path based upon a link’s utilization is not possible without some sort of hacked up solution (EEM, TCL, packet marking with policy based routing, etc). PfR will moving these traffic flows for us. As the primary Ethernet link’s utilization rises above the 75% utilization threshold, PfR will start to look for an alternate path to route traffic flows across in order to bring down the utilization on the Ethernet link. In our scenario, PfR will automatically use NetFlow, the interface’s utilization, along with active probing with IP SLA to determine the availability of an alternate path. Knowledge of NetFlow and IP SLA is not needed as PfR manages the configuration of these technologies.
Now that we know the problem we are trying to solve let’s start looking at PfR itself. For PfR, we have the concept of a Master Controller (MC) and Border Router/Routers (BR). The MC is the centralized decision maker in the PfR network. It is responsible for controlling the BRs and acts as a central location to store and analyze data collected from the BRs. The MC does not need to be in the path of the traffic flows but does need basic IP reachability to the BRs. Often in real-world medium and large-scale PfR deployments, the MC will be a dedicated router. Traditionally, the BRs are the edge routers with links to external networks (i.e. the Internet). This is where the term Optimized Edge Routing (OER) came from as we are optimizing the routing at the edge of our network but as we will see later, the added ability in the newer IOS versions for routing based upon network and/or application performance, Cisco renamed it Performance Routing (PfR). This is why today, PfR is suited for used in enterprise networks as we will see in future blog posts. The BRs are in the traffic flow paths and are responsible for collecting the NetFlow data and IP SLA probe results along with influencing the traffic flow paths by executing policy change instructions issued by the MC.
The communication between the MC and BRs uses an authenticated TCP connection on port 3949. The BR is responsible for initiating the TCP connection. In the case of a firewall or any traffic filters between the MC and BR, TCP port 3949 needs to be open in the path of the BR towards the MC. The default TCP port can be changed and the default interface used for the connection (based upon the outbound interface used to route to the MC) can be changed. The password used for authentication is done similar to how EIGRP and RIP handle authentication in IPv4, and that is through the use of key chains. Being that PfR is a path selection technology we will need to define at least one internal and two external interfaces. In our PfR setup along with the requirement of having a MC and BR will need to have at least one internal and two external interfaces on the BR. These interfaces will be defined on the MC.
NOTE
The two external interfaces are not required to be located on the same BR. One external interface could be located on one BR, and additional external interfaces could be located on another BR or even several BRs. External interfaces are not required to be physical interface and can be logical interfaces (Tunnels, Dialer, etc).
The PfR will enable NetFlow on the BR’s interfaces to allow for the collection of traffic statistics to be sent back to the MC. For our first basic example, we will have the MC and BR located on the same router. Below is the diagram for this scenario:
Here is the relevant configuration for this scenario prior to configuring PfR. To ease the creation of congestion, the T1 has been clocked down to 64k, and the Ethernet link is being shaped to 256k. You will notice the floating static route pointing out the Serial link in the event the Ethernet interface goes down. This additional static route is what PfR will term a parent route for the Serial link. Since it’s not important at this point, the concept of a parent route will be covered at the end of the blog post. We just need to understand that an additional route that encompasses our traffic is needed over the alternate external interface. Lastly, the load interval for the interfaces has been set to 30 seconds contrary to the default of 300 seconds. This allows PfR to react quicker to changes in the load (throughput) of the interface.
Rack1R3#show run Building configuration... Current configuration : 2236 bytes ! version 12.4 ! hostname Rack1R3 ! policy-map 256_SHAPE class class-default shape average 256000 8000 0 ! interface Loopback0 ip address 150.1.3.3 255.255.255.255 ! interface FastEthernet0/0 description Internal ip address 204.12.1.3 255.255.255.0 ! interface FastEthernet0/1 description External bandwidth 256 ip address 192.10.1.3 255.255.255.0 load-interval 30 service-policy output 256_SHAPE ! interface Serial1/0 encapsulation frame-relay load-interval 30 ! interface Serial1/0.1 point-to-point description External bandwidth 64 ip address 192.10.2.3 255.255.255.0 frame-relay interface-dlci 301 ! ip route 0.0.0.0 0.0.0.0 192.10.1.1 ip route 0.0.0.0 0.0.0.0 192.10.2.1 5 ! end Rack1R3#show ip route Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route Gateway of last resort is 192.10.1.1 to network 0.0.0.0 C 204.12.1.0/24 is directly connected, FastEthernet0/0 C 192.10.1.0/24 is directly connected, FastEthernet0/1 C 192.10.2.0/24 is directly connected, Serial1/0.1 150.1.0.0/32 is subnetted, 3 subnets O 150.1.7.7 [110/2] via 204.12.1.7, 2d22h, FastEthernet0/0 O 150.1.6.6 [110/2] via 204.12.1.6, 2d23h, FastEthernet0/0 C 150.1.3.3 is directly connected, Loopback0 S* 0.0.0.0/0 [1/0] via 192.10.1.1 Rack1R3#
NOTE
With static routes it’s recommended in production environments to use IP SLA and Enhanced Object Tracking to monitor the primary static route’s next-hop availability if the interface’s state does not indicate availability of the next-hop. This is common when a layer 2 switch is located between the local router and the next-hop.
We will start with the BR configuration. First, we need to define a key chain in the global configuration:
Rack1R3(config)#key chain OER Rack1R3(config-keychain)#key 1 Rack1R3(config-keychain-key)#key-string CISCO Rack1R3(config-keychain-key)#exit Rack1R3(config-keychain)#exit Rack1R3(config)#exit Rack1R3#show run | section key chain key chain OER key 1 key-string CISCO Rack1R3#
The key chain name is arbitrary and in this case, we used the name OER. We used the key numbered 1 with a key string of CISCO. For this example since the MC and BR will be located on a single router the same key chain will be used. In fact, if you change the key chain name used for the MC under the BR configuration sub-mode it will automatically change it for the BR under the MC configuration sub-mode. Some sort of idiot-proofing in the IOS Next we will configure the actual BR. On the BR we will configure the interface used to source the TCP connection off of and the IP address of the MC along with the key chain used to authenticate the session. The BR configuration is done in the OER border configuration sub-mode. To enter this mode type oer border in the global configuration.
Rack1R3(config)#oer border Rack1R3(config-oer-br)#local Loopback0 Rack1R3(config-oer-br)#master 150.1.3.3 key-chain OER Rack1R3(config-oer-br)#exit Rack1R3(config)#exit Rack1R3#show run | section oer border oer border local Loopback0 master 150.1.3.3 key-chain OER Rack1R3#
The local command is used to control the interface the TCP session is sourced off of. This commonly is a loopback but could be any interface that has reachability to the MC. The master command is used to define the IP address of the master along with the key chain. The only other requirement for the BR is that CEF is enabled, which should be by default but remember to check that it is enabled when troubleshooting a PfR issue.
NOTE
As of IOS version 15.0 the oer keyword is changed to pfr and the oer keyword will eventually be removed from the IOS.Rack1R1(config)#do show version | include Version 15.1 Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(3)T2, RELEASE SOFTWARE (fc1) Rack1R1(config)#oer ? border Enter PfR border router configuration submode master Enter PfR master controller configuration submode Rack1R1(config)#pfr ? border Enter PfR border router configuration submode master Enter PfR master controller configuration submode Rack1R1(config)#pfr border Rack1R1(config-pfr-br)#exit Rack1R1(config)#oer border Rack1R1(config-pfr-br)#
Now we will configure the MC. As with the BR configuration being done in the config-oer-br configuration sub-mode, the MC will be done in similar configuration sub-mode. We will start off by entering the MC configuration sub-mode by issuing the oer master command from the global configuration. Next we will define the BR’s IP address, key chain, internal and external interfaces.
Rack1R3(config)#oer master Rack1R3(config-oer-mc)#border 150.1.3.3 key-chain OER Rack1R3(config-oer-mc-br)#interface FastEthernet0/0 internal Rack1R3(config-oer-mc-br)#interface Serial1/0.1 external Rack1R3(config-oer-mc-br-if)#interface FastEthernet0/1 external Rack1R3(config-oer-mc-br-if)#exit Rack1R3(config-oer-mc-br)#exit Rack1R3(config-oer-mc)#exit Rack1R3(config)#exit Rack1R3#show run | section oer master oer master ! border 150.1.3.3 key-chain OER interface FastEthernet0/0 internal interface Serial1/0.1 external interface FastEthernet0/1 external Rack1R3#
We can now verify that the MC and BR processes on the router are communicating:
Rack1R3#show oer master OER state: ENABLED and ACTIVE Conn Status: SUCCESS, PORT: 3949 Version: 2.2 Number of Border routers: 1 Number of Exits: 2 Number of monitored prefixes: 0 (max 5000) Max prefixes: total 5000 learn 2500 Prefix count: total 0, learn 0, cfg 0 PBR Requirements met Nbar Status: Inactive Border Status UP/DOWN AuthFail Version 150.1.3.3 ACTIVE UP 00:00:52 0 2.2 Global Settings: max-range-utilization percent 20 recv 0 mode route metric bgp local-pref 5000 mode route metric static tag 5000 trace probe delay 1000 no logging exit holddown time 60 secs, time remaining 0 Default Policy Settings: backoff 300 3000 300 delay relative 50 holddown 300 periodic 0 probe frequency 56 number of jitter probe packets 100 mode route observe mode monitor both mode select-exit good loss relative 10 jitter threshold 20 mos threshold 3.60 percent 30 unreachable relative 50 resolve delay priority 11 variance 20 resolve range priority 12 variance 0 resolve utilization priority 13 variance 20 Learn Settings: current state : DISABLED time remaining in current state : 0 seconds no throughput no delay no inside bgp no protocol monitor-period 5 periodic-interval 120 aggregation-type prefix-length 24 prefixes 100 expire after time 720 Rack1R3#show oer border OER BR 150.1.3.3 ACTIVE, MC 150.1.3.3 UP/DOWN: UP 00:01:00, Auth Failures: 0 Conn Status: SUCCESS OER Netflow Status: ENABLED, PORT: 3949 Version: 2.2 MC Version: 2.2 Exits Fa0/0 INTERNAL Fa0/1 EXTERNAL Se1/0.1 EXTERNAL Rack1R3#show oer master border detail Border Status UP/DOWN AuthFail Version 150.1.3.3 ACTIVE UP 01:25:10 0 2.2 Fa0/0 INTERNAL UP Se1/0.1 EXTERNAL UP Fa0/1 EXTERNAL UP External Capacity Max BW BW Used Load Status Exit Id Interface (kbps) (kbps) (kbps) (%) --------- -------- ------ ------- ------- ------ ------ Se1/0.1 Tx 64 48 0 0 UP 2 Rx 64 0 0 Fa0/1 Tx 256 192 0 0 UP 1 Rx 256 0 0 Rack1R3#
NOTE
For the PfR versions, there is a major release and minor release number. For example, PfR v2.2 in IOS 12.4T or PfR v3.0 in IOS 15.1. The “2.x” and “3.x” are the major version numbers and “x.2″ and “x.0″ are the minor version numbers. The PfR major version number must match between the MC and BRs. Additionally the minor release version on the MC must be equal to or greater than the BR’s minor release version. The version used can be determined by issuing the show oer master on the MC and show oer border on the BR.
Now that we have our MC and BR defined along with our internal and external interfaces, we need to configure the MC to start monitoring the Ethernet link’s throughput rate so that traffic can be moved to the Serial link once the 75% threshold has been breached. To do this we need to enter the learn configuration sub-mode under the MC configuration sub-mode. If you look back at the output from the show oer master command, you will notice that the default state for learning phase is disabled. Once we are under the learning configuration sub-mode we will configure the MC to starting monitoring the throughput of the traffic flows from the internal interfaces out to the external interfaces.
Rack1R3(config-oer-mc)#learn Rack1R3(config-oer-mc-learn)#throughput Rack1R3(config-oer-mc-learn)#do sho oer master | begin Learn Learn Settings: current state : STARTED time remaining in current state : 358 seconds throughput no delay no inside bgp no protocol monitor-period 5 periodic-interval 120 aggregation-type prefix-length 24 prefixes 100 expire after time 720 Rack1R3(config-oer-mc-learn)#
The default interface utilization threshold is 75% when monitoring throughput. This can be changed under the interface configuration sub-mode for a particular BR (config-oer-mc-br-if). The value can be entered as either a percentage or absolute value. The configuration flow is as follows:
Rack1R3(config)#oer master Rack1R3(config-oer-mc)#border 150.1.3.3 key-chain OER Rack1R3(config-oer-mc-br)#interface FastEthernet0/1 external Rack1R3(config-oer-mc-br-if)#max-xmit-utilization ? absolute Specify the utilization as an absolute value percentage Specify the utilization as a percentage of the exit's bandwidth
The MC can generate log messages containing information in regard to its operation. By default, this behavior is disabled but can be enabled by issuing the logging command under the OER master configuration sub-mode. This will help us get a better understanding of what PfR is doing without the need for show commands or enabling debugging:
Rack1R3(config)#oer master Rack1R3(config-oer-mc)#logging
By default the MC is in a route control mode of observe. This means monitoring and reporting will occur but no policy changes will be issued by the MC to the BRs. In this mode we can view what the MC is observing and what actions it would take assuming it wasn’t in observe mode. Below are the log messages from the MC when the 75% threshold is breached and the MC is in observe mode.
Oct 31 11:50:05: %OER_MC-5-NOTICE: Range OOP BR 150.1.3.3, i/f Fa0/1, percent 100. Other BR 150.1.3.3, i/f Se1/0.1, percent 0 Oct 31 11:50:05: %OER_MC-5-NOTICE: Load OOP BR 150.1.3.3, i/f Fa0/1, load 256 policy 192 Oct 31 11:50:05: %OER_MC-5-NOTICE: Exit 150.1.3.3 intf Fa0/1 OOP, Tx BW 256, Rx BW 256, Tx Load 100, Rx Load 100 Oct 31 11:50:27: %OER_MC-5-NOTICE: NO route change, Observe mode, Prefix 114.0.1.0/24, BR 150.1.3.3, i/f Se1/0.1, Reason Delay, OOP reason Range
For our scenario, we want the BR to actually move traffic flows by injecting a more specific static route over the T1 link when the Ethernet’s link utilization threshold of 75% is breached. In order to do this we need PfR to determine what is called the traffic classes from the traffic flows that are flowing through BRs from the internal interface to an external interface. When OER was first introduced in IOS 12.3(8)T only layer 3 network prefix based traffic classes were supported. In newer IOS versions, we can define traffic classes based upon the layer 4 port number, DSCP value or even application using NBAR. In this scenario, we will use automatically learned layer 3 prefix traffic classes but in future blog posts, we will manually define the traffic flows that we are interested in PfR monitoring.
Since we are going to allow PfR to learn the traffic classes, these classes will be based upon the Netflow Top Talkers and then aggregated into /24 traffic classes. By default, PfR aggregates traffic flows into /24 traffic classes before sending them to the MC. The level of aggregation can be changed using the aggregation-type prefix-length <1-32> command in the learn configuration sub-mode. Once received by the MC these traffic classes are what is called Monitored Traffic Classes or MTC for short. We will be able to view these by using the show oer master traffic-class command. What this /24 aggregation means for our scenario is that when we have two traffic flows that are not in the same /24 address space (i.e. traffic destined for 114.0.0.1 and traffic destined for 114.0.1.1) the BR will aggregate these into two traffic classes: 114.0.0.0/24 and 114.0.1.0/24. Once the threshold is breached the traffic class will be considered to be what is called Out of Policy (OOP). When this happens the MC will try to move the one of the traffic classes to the T1 link by injecting a static route for the specific traffic class (i.e. 114.0.1.0/24) out the Serial1/0.1 external interface.
To have PfR automatically learn the traffic classes and alter the static routing for our scenario, we will need to enable the learning of the prefixes and trigger the route policy change based upon the external interfaces’ throughput. We could base the trigger upon delay, packet loss and/or reachability but for this simplest of cases, we’re just going to use throughput. Additionally, we will need to change the MC’s route control policy settings from observe (mode route observe) to route control (mode route control) so that the MC can instruct the BR to inject static routes allowing it to move traffic from the Ethernet link to the T1 link. This mode can be seen using the show oer master command.
Rack1R3#show oer master | include ^Default|mode route control Default Policy Settings: mode route control Rack1R3#conf t Enter configuration commands, one per line. End with CNTL/Z. Rack1R3(config)#oer master Rack1R3(config-oer-mc)#mode route control Rack1R3(config-oer-mc)#learn Rack1R3(config-oer-mc-learn)#throughput Rack1R3(config-oer-mc-learn)#exit Rack1R3(config-oer-mc)#exit Rack1R3(config)#exit
NOTE
The MC will learn up to 100 of these traffic classes by default. This can be changed by using the prefixes <1-100000> command in the learn configuration sub-mode. Before IOS 12.2(15)T the IOS was limited to a maximum of 5000 prefixes, but this limitation was bumped up to 100000 in the more recent IOS versions.
Before we actually start to generate traffic flows for testing, let’s take a look at our final configuration (relevant commands):
Rack1R3#show run Building configuration... Current configuration : 2283 bytes ! version 12.4 ! hostname Rack1R3 ! ip cef ! key chain OER key 1 key-string CISCO ! oer master logging ! border 150.1.3.3 key-chain OER interface FastEthernet0/0 internal interface Serial1/0.1 external interface FastEthernet0/1 external ! learn throughput mode route control ! ! oer border local Loopback0 master 150.1.3.3 key-chain OER ! policy-map 256_SHAPE class class-default shape average 256000 8000 0 ! interface Loopback0 ip address 150.1.3.3 255.255.255.255 ! interface FastEthernet0/0 description Internal ip address 204.12.1.3 255.255.255.0 ! interface FastEthernet0/1 description External bandwidth 256 ip address 192.10.1.3 255.255.255.0 load-interval 30 service-policy output 256_SHAPE ! interface Serial1/0 encapsulation frame-relay load-interval 30 ! interface Serial1/0.1 point-to-point description External bandwidth 64 ip address 192.10.2.3 255.255.255.0 frame-relay interface-dlci 301 ! ip route 0.0.0.0 0.0.0.0 192.10.1.1 ip route 0.0.0.0 0.0.0.0 192.10.2.1 5 ! end
Next we’re going to start with the actual testing. First off, we’re going to reset the MC’s state by issuing the clear oer master * command. Next we will generate a large ICMP flow by pinging from behind R3 (204.12.1.6) out to 114.0.0.1. The packet size is set to 1400, repeat 10000000 and timeout is set to 0. This will cause the BR’s external Ethernet link to exceed the 75% threshold of 192k (256k * .75). Next we will telnet to 114.0.1.1 from behind R3 (204.12.1.5). This traffic flow will be aggregated to 114.0.1.0/24 and be moved by the MC, using a static route injected on the BR, to the T1 link.
Rack1R3#clear oer master * Rack1R3# Oct 31 17:55:22: %OER_MC-5-NOTICE: Uncontrol prefixes, Clear all Oct 31 17:55:22: %OER_MC-5-NOTICE: BR 150.1.3.3 DOWN Oct 31 17:55:22: %OER_MC-5-NOTICE: BR 150.1.3.3 IF Fa0/0 Unverified Oct 31 17:55:22: %OER_MC-5-NOTICE: BR 150.1.3.3 IF Se1/0.1 Unverified Oct 31 17:55:22: %OER_MC-5-NOTICE: BR 150.1.3.3 IF Fa0/1 Unverified Oct 31 17:55:22: %OER_MC-5-NOTICE: MC Inactive Oct 31 17:55:23: %OER_MC-5-NOTICE: Uncontrol prefixes, Exit down, BR 150.1.3.3 i/f Se1/0.1 Oct 31 17:55:23: %OER_MC-5-NOTICE: Uncontrol prefixes, Exit down, BR 150.1.3.3 i/f Fa0/1 Oct 31 17:55:23: %OER_MC-5-NOTICE: Uncontrol prefixes, Clear Application all Oct 31 17:55:23: %OER_MC-5-NOTICE: Uncontrol prefixes, Clear prefix all Rack1R3# Oct 31 17:55:28: %OER_MC-5-NOTICE: BR 150.1.3.3 UP Oct 31 17:55:29: %OER_MC-5-NOTICE: BR 150.1.3.3 IF Fa0/0 UP Oct 31 17:55:29: %OER_MC-5-NOTICE: BR 150.1.3.3 IF Se1/0.1 UP Oct 31 17:55:29: %OER_MC-5-NOTICE: BR 150.1.3.3 Active Oct 31 17:55:29: %OER_MC-5-NOTICE: BR 150.1.3.3 IF Fa0/1 UP Oct 31 17:55:29: %OER_MC-5-NOTICE: MC Active Rack1R3# Oct 31 17:55:59: %OER_MC-5-NOTICE: Prefix Learning STARTED Rack1R3# Oct 31 17:56:28: %OER_MC-5-NOTICE: Range OOP BR 150.1.3.3, i/f Fa0/1, percent 100. Other BR 150.1.3.3, i/f Se1/0.1, percent 0 Oct 31 17:56:28: %OER_MC-5-NOTICE: Load OOP BR 150.1.3.3, i/f Fa0/1, load 256 policy 192 Oct 31 17:56:28: %OER_MC-5-NOTICE: Exit 150.1.3.3 intf Fa0/1 OOP, Tx BW 256, Rx BW 8, Tx Load 100, Rx Load 3 Oct 31 18:00:49: %OER_MC-5-NOTICE: Range OOP BR 150.1.3.3, i/f Fa0/1, percent 100. Other BR 150.1.3.3, i/f Se1/0.1, percent 0 Oct 31 18:00:49: %OER_MC-5-NOTICE: Load OOP BR 150.1.3.3, i/f Fa0/1, load 256 policy 192 Oct 31 18:00:49: %OER_MC-5-NOTICE: Exit 150.1.3.3 intf Fa0/1 OOP, Tx BW 256, Rx BW 8, Tx Load 100, Rx Load 3 Rack1R3# Oct 31 18:00:59: %OER_MC-5-NOTICE: Prefix Learning WRITING DATA Rack1R3# Oct 31 18:01:09: %OER_MC-5-NOTICE: Range OOP BR 150.1.3.3, i/f Fa0/1, percent 100. Other BR 150.1.3.3, i/f Se1/0.1, percent 0 Oct 31 18:01:09: %OER_MC-5-NOTICE: Load OOP BR 150.1.3.3, i/f Fa0/1, load 256 policy 192 Oct 31 18:01:09: %OER_MC-5-NOTICE: Exit 150.1.3.3 intf Fa0/1 OOP, Tx BW 256, Rx BW 8, Tx Load 100, Rx Load 3 Rack1R3# Oct 31 18:01:26: %OER_MC-5-NOTICE: Discovered Exit for Prefix 114.0.1.0/24, BR 150.1.3.3, i/f Fa0/1 Oct 31 18:01:26: %OER_MC-5-NOTICE: Discovered Exit for Prefix 114.0.0.0/24, BR 150.1.3.3, i/f Fa0/1 Rack1R3# Oct 31 18:01:29: %OER_MC-5-NOTICE: Range OOP BR 150.1.3.3, i/f Fa0/1, percent 100. Other BR 150.1.3.3, i/f Se1/0.1, percent 0 Oct 31 18:01:29: %OER_MC-5-NOTICE: Load OOP BR 150.1.3.3, i/f Fa0/1, load 256 policy 192 Oct 31 18:01:29: %OER_MC-5-NOTICE: Exit 150.1.3.3 intf Fa0/1 OOP, Tx BW 256, Rx BW 8, Tx Load 100, Rx Load 3 Rack1R3# Oct 31 18:02:29: %OER_MC-5-NOTICE: Uncontrol Prefix 114.0.0.0/24, Couldn't find the best exit Oct 31 18:02:29: %OER_MC-5-NOTICE: Uncontrol Prefix 114.0.0.0/24, Couldn't choose exit in prefix timeout Oct 31 18:02:29: %OER_MC-5-NOTICE: Range OOP BR 150.1.3.3, i/f Fa0/1, percent 100. Other BR 150.1.3.3, i/f Se1/0.1, percent 0 Oct 31 18:02:29: %OER_MC-5-NOTICE: Load OOP BR 150.1.3.3, i/f Fa0/1, load 256 policy 192 Oct 31 18:02:29: %OER_MC-5-NOTICE: Exit 150.1.3.3 intf Fa0/1 OOP, Tx BW 256, Rx BW 8, Tx Load 100, Rx Load 3 Oct 31 18:02:29: %OER_MC-5-NOTICE: Route changed Prefix 114.0.1.0/24, BR 150.1.3.3, i/f Se1/0.1, Reason Delay, OOP Reason Range Rack1R3#
From the output of the MC logging, we can see that at 18:02:29 a static route is injected into the BR’s routing table for the 114.0.1.0/24 MTC which took roughly 7 minutes from the time we restarted the MC (clear oer master *). This is the MTC for the telnet traffic flow from 204.12.1.5 to 114.0.1.1. Later, we will speed this process up and even configure PfR to switch within a matter of just a few seconds.
Rack1R3#show oer master traffic-class OER Prefix Statistics: Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms), P - Percentage below threshold, Jit - Jitter (ms), MOS - Mean Opinion Score Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million), E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable U - unknown, * - uncontrolled, + - control more specific, @ - active probe all # - Prefix monitor mode is Special, & - Blackholed Prefix % - Force Next-Hop, ^ - Prefix is denied DstPrefix Appl_ID Dscp Prot SrcPort DstPort SrcPrefix Flags State Time CurrBR CurrI/F Protocol PasSDly PasLDly PasSUn PasLUn PasSLos PasLLos EBw IBw ActSDly ActLDly ActSUn ActLUn ActSJit ActPMOS ActSLos ActLLos -------------------------------------------------------------------------------- 114.0.0.0/24 N defa N N N N DEFAULT* 21 150.1.3.3 Fa0/1 U U U 0 0 0 0 8204 0 U U 0 1000000 N N N N 114.0.1.0/24 N defa N N N N INPOLICY 0 150.1.3.3 Se1/0.1 STATIC 20 20 0 0 0 0 1 1 U U 0 0 N N N N Rack1R3# Rack1R3#show ip route static 114.0.0.0/24 is subnetted, 1 subnets S 114.0.1.0 [1/0] via 192.10.2.1 //将更精确的静态路由注入路由表 S* 0.0.0.0/0 [1/0] via 192.10.1.1 Rack1R3# Rack1R3#show oer border routes static Flags: C - Controlled by oer, X - Path is excluded from control, E - The control is exact, N - The control is non-exact Flags Network Parent Tag CE 114.0.1.0/24 0.0.0.0/0 5000 Rack1R3#
NOTE
The static route injected on the BR will not show up in the running configuration which means it will not be saved upon a reload. This is done to ensure if ever communication is lost between the MC and BR that any policy changes issued by the MC cannot be saved and are quickly removed.Rack1R3#show run | include ip route ip route 0.0.0.0 0.0.0.0 192.10.1.1 ip route 0.0.0.0 0.0.0.0 192.10.2.1 5
We can now see that everything worked as expected and the traffic destined for the 114.0.1.0/24 prefix is being routed across the T1 using the static route injected by PfR over the less specific static default route. As mentioned earlier PfR uses NetFlow for collecting statistics about the traffic flows. This can be seen by using the show ip cache flow command.
Rack1R3#show ip cache flow IP packet size distribution (65263711 total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .000 .000 .000 .000 .999 .000 .000 .000 .000 .000 .000 IP Flow Switching Cache, 278544 bytes 3 active, 4093 inactive, 3747 added 174735 ager polls, 0 flow alloc failures Active flows timeout in 1 minutes Inactive flows timeout in 15 seconds IP Sub Flow Cache, 34056 bytes 8 active, 1016 inactive, 6691 added, 3747 added to flow 0 alloc failures, 0 force free 1 chunk, 20 chunks added last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) -------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-Telnet 385 0.0 108 143 0.0 31.3 7.3 ICMP 2379 0.0 27404 1463 102.5 59.8 0.4 Total: 2764 0.0 23602 1462 102.6 55.9 1.4 SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa0/0 204.12.1.6 Fa0/1 114.0.0.1 01 0000 0800 26K Fa0/1 114.0.1.1 Fa0/0 204.12.1.5 06 0017 3893 174 Fa0/0 204.12.1.5 Se1/0.1 114.0.1.1 06 3893 0017 256 Rack1R3#
It was mentioned earlier that by default PfR uses IP SLA along with NetFlow. Although we will go into more detail on IP SLA with PfR later, how PfR used ICMP echo probes to assist with path selection for our scenario can be seen below:
Rack1R3#show oer master active-probes OER Master Controller active-probes Border = Border Router running this Probe State = Un/Assigned to a Prefix Prefix = Probe is assigned to this Prefix Type = Probe Type Target = Target Address TPort = Target Port How = Was the probe Learned or Configured N - Not applicable The following Probes exist: State Prefix Type Target TPort How Codec Assigned 114.0.1.0/24 echo 114.0.1.1 N Lrnd N Assigned 114.0.0.0/24 echo 114.0.0.1 N Lrnd N The following Probes are running: Border State Prefix Type Target TPort 150.1.3.3 ACTIVE 114.0.1.0/24 echo 114.0.1.1 N 150.1.3.3 ACTIVE 114.0.1.0/24 echo 114.0.1.1 N 150.1.3.3 ACTIVE 114.0.0.0/24 echo 114.0.0.1 N 150.1.3.3 ACTIVE 114.0.0.0/24 echo 114.0.0.1 N Rack1R3#show oer border active-probes OER Border active-probes Type = Probe Type Target = Target IP Address TPort = Target Port Source = Send From Source IP Address Interface = Exit interface Att = Number of Attempts Comps = Number of completions N - Not applicable Type Target TPort Source Interface Att Comps DSCP echo 114.0.1.1 N 192.10.1.3 Fa0/1 1 1 0 echo 114.0.1.1 N 192.10.2.3 Se1/0.1 1 1 0 echo 114.0.0.1 N 192.10.1.3 Fa0/1 1 0 0 echo 114.0.0.1 N 192.10.2.3 Se1/0.1 1 0 0
Although this completes our relatively simple scenario, I will like to come back to the concept of the parent route briefly covered earlier. As mentioned the MC will, by default, try to aggregate traffic into /24 prefixes (traffic classes). In order for the MC to instruct the BR to inject a new static route for a particular flow out an alternate external interface, a parent route needs to already exist pointing out the alternate external interface.//替代路径也必须是已存在的路由条目,不能凭空生成,因此预配置静态路由的目的也基于此。 In our scenario, we have a floating static route pointing out the Serial1/0.1 interface. Although this static route isn’t in the routing table due to the higher administrative distance, it is still needed for PfR to use the alternate interface. PfR will need to verify that the interface is useable and reachability can be obtained across the alternate external interface. In our case, PfR used IP SLA because by default PfR operates in a prefix monitoring mode of what is termed as both. The term both is referring to both active and passive monitoring. Active monitoring means that PfR will generate test traffic (probes), ICMP ECHOs in our case, to verify reachability across the alternate path out the Serial1/0.1 interface. As we will see in future blog posts, we can specify the type of test traffic we want generated. For instance, if we are using PfR to optimize VoIP traffic in our internal network, we could use VoIP voice packets with IP SLA for the active probing.
So the point here is to remember that a parent route, which is defined as a route that is equal to or less specific than the monitored traffic class (MTC), is needed but does not actually have to be in the routing table when static routes are used. Just being in the global configuration is adequate to be considered a parent route.
Valid due to the fact a parent route does exist for the alternate external interface:
Rack1R3#show run | include ip route ip route 0.0.0.0 0.0.0.0 192.10.1.1 ip route 0.0.0.0 0.0.0.0 192.10.2.1 5 Rack1R3#show ip route static S* 0.0.0.0/0 [1/0] via 192.10.1.1 Rack1R3#
Invalid due to the fact a parent route does not exist for the alternate external interface:
Rack1R3#show run | include ip route ip route 0.0.0.0 0.0.0.0 192.10.1.1 Rack1R3#show ip route static S* 0.0.0.0/0 [1/0] via 192.10.1.1 Rack1R3#
Although this blog post my seem long there are a lot of details that were left out but will be covered in future posts on PfR. In the next blog post we will define our own traffic classes based upon the DSCP values and applications for PfR to monitor along with a PfR load balancing scenario.