The Cisco ASR 1000 is Cisco's new enterprise and service provider router.  With this new router comes new caveats and platform restrictions.  One of which is the interface counters.

The 'show interface' of the ASR CLI is somewhat deceiving when talking about utilization numbers.  On an Ethernet interface, you must account for 38 bytes of Ethernet overhead when you talk about utilization.  That is broken up as follows:


Inter Frame Gap          12 bytes
MAC Preamble (w/ SoF)     8 bytes
MAC Destination           6 bytes
MAC Source                6 bytes
Length                    2 bytes
CRC                       4 bytes
---------------------------------
Total Overhead           38 bytes


If you take a 1500 byte IP packet, you need to add 38 bytes for the Layer 2 overhead.  The same applies for a minimum sized IP packet, which is 46 bytes.  You're left with frame sizes of 1538 and 84 bytes.  Because the overhead is fixed, but the IP packet/payload is variable (in the real world), the overhead ratio is variable.

A 46 byte packet takes up 84 bytes on the wire, we have 45.2% of Ethernet overhead
A 1500 byte packet takes up 1538 bytes on the wire, we have 2.5% of Ethernet overhead

In the Cisco CLI, the 'show interface' only accounts for part of the Layer 2 overhead.  Specifically: Source MAC, Destination MAC, and Length.  That totals 14 bytes, not 38 bytes!  What this means is that the CLI will not report the actual link utilization (including all L2 overhead), but only a fraction of the true value.

Here is the sample output from a Cisco ASR 1002, with a traffic generator attached to two of the built in GIG-E interfaces.  The traffic generator is setup for a unidirectional flow from one port to another, running at wire rate, with 46 byte IP packets.  The output was taken about 9 minutes into the test, which should have given all of the counters enough time to stabilize, I'm specifically thinking about the '30 second' counters.

ASR#show version | inc XE
Cisco IOS Software, IOS-XE Software (PPC_LINUX_IOSD-ADVENTERPRISEK9-M), Version 12.2(33)XND1, RELEASE SOFTWARE (fc1)

ASR#sh int gi 0/0/2
GigabitEthernet0/0/2 is up, line protocol is up
  Hardware is 4XGE-BUILT-IN, address is 0046.0b44.8202 (bia 0046.0b44.8202)
  Description: Test
  Internet address is 10.0.0.1/27
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 182/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  Full Duplex, 1000Mbps, link type is auto, media type is SX
  output flow-control is off, input flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:09:44, output 00:00:41, output hang never
  Last clearing of "show interface" counters 00:11:38
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 714272000 bits/sec, 1488068 packets/sec
     2 packets input, 120 bytes, 0 no buffer
     Received 2 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     844592585 packets output, 50675554817 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out


Based on the 30 second input rate value, you would think you have 286 mbps of bandwidth to spare.  That 286 mbps is actually not available, it is the overhead used by that additional 24 bytes of overhead!

Because the traffic generator is sending in a fixed packet size, 46 bytes, and the router is telling us the output packet rate, we can do some simple math to figure out the real throughput rate (accounting for all of the Ethernet overhead).

The Layer 3 (IP) packet rate, as reported by the router, is 1488068.  A 46 byte IP packet, with the 38 bytes of overhead (per packet), is 84 bytes - or 672 bits.  If you multiply those two figures together, we get the actual bps value of the interface: 999,981,696

That interface is effectively 100% utilized, despite the deceiving value in the CLI reporting that it has 286 mbps of throughput left.

An 84 byte frame is the worst case scenario, here is a breakdown with the best case scenario, a 1535 byte frame.

ASR#sh int gi 0/0/2
GigabitEthernet0/0/2 is up, line protocol is up
  Hardware is 4XGE-BUILT-IN, address is 0046.0b44.8202 (bia 0046.0b44.8202)
  Description: Test
  Internet address is 10.0.0.1/27
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 250/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  Full Duplex, 1000Mbps, link type is auto, media type is SX
  output flow-control is off, input flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:03:52, output 00:03:52, output hang never
  Last clearing of "show interface" counters 00:03:32
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 984254000 bits/sec, 81270 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     17881974 packets output, 27073308636 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

The Layer 3 (IP) packet rate, as reported by the router, is 81270.  A 1500 byte IP packet, with the 38 bytes of overhead (per packet), is 1538 bytes - or 12304 bits.  If you multiply those two figures together, we get the actual bps value of the interface: 999,946,080

That interface is effectively 100% utilized, despite the deceiving value in the CLI reporting that it has 15 mbps of throughput left.  Because the Ethernet overhead to payload ratio is much better with full size packets, the counters are much more accurate, but still not 100%.

In production environments, running link utilizations above 50%, it's important to know what your average packet size is so you can accurately know when links are fully saturated.