利用pktgen发包工具做性能测试

专业的发包仪器非常不错，比如思博伦通信Smartbits测试设备，但唯一的缺点就是太贵，而这个唯一的缺点又不是那么容易克服。还好，伟大的Linux为我们提供了一个先进的发包工具pktgen（http://www.linuxfoundation.org/collaborate/workgroups/networking/pktgen、http://lxr.linux.no/#linux+v2.6.38.8/Documentation/networking/pktgen.txt、ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/），这个工具以内核模块的形式存在，理论上性能应该比同等运行在应用层的工具性能要好，而且还是所谓的多核支持（的确是实打实），下面测试一下（临时用自己家里电脑上的虚拟机搭的环境，以便能够虚拟出需要的多个网卡来，不在意测试结果，而是这个测试过程）。

1，首先加载这个模块，当然需要先确定已经编译了它，加载成功后就会有对应的/proc接口，pktgen是每一个cpu绑定一个内核线程，我的虚拟机有4个cpu，所以这里可以看到4个kpktgend_*文件：

1 2 3 4 5 6 7 8 9 10 11 12 13

[root@localhost ~]# uname -a Linux localhost.localdomain 2.6.38.8 #4 SMP Mon Oct 31 20:49:48 CST 2011 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# cat /usr/src/linux-2.6.38.8/.config | grep CONFIG_NET_PKTGEN CONFIG_NET_PKTGEN=m [root@localhost ~]# insmod /usr/src/linux-2.6.38.8/net/core/pktgen.ko [root@localhost ~]# cat /proc/cpuinfo | grep processor processor : 0 processor : 1 processor : 2 processor : 3 [root@localhost ~]# ls /proc/net/pktgen/ kpktgend_0 kpktgend_1 kpktgend_2 kpktgend_3 pgctrl [root@localhost ~]#

2，虚拟机一共有3个网口，拿其中的2个网口（eth5/eth6）用于pktgen测试，设置ip地址，修改速率，：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

[root@localhost pktgen]# ifconfig eth5 eth5 Link encap:Ethernet HWaddr 00:0C:29:97:9B:B4 inet addr:192.168.1.95 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe97:9bb4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15044140 errors:0 dropped:0 overruns:0 frame:0 TX packets:14210498 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:903846363 (861.9 MiB) TX bytes:852636332 (813.1 MiB) [root@localhost pktgen]# ifconfig eth6 eth6 Link encap:Ethernet HWaddr 00:0C:29:97:9B:BE inet addr:192.168.1.96 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe97:9bbe/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14256340 errors:0 dropped:0 overruns:0 frame:0 TX packets:14998657 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:856589260 (816.9 MiB) TX bytes:899925597 (858.2 MiB) [root@localhost pktgen]# ethtool -s eth5 autoneg off speed 1000 duplex full [root@localhost pktgen]# ethtool -s eth6 autoneg off speed 1000 duplex full [root@localhost pktgen]# ethtool eth5 Settings  for  eth5: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) Link detected: yes [root@localhost pktgen]# ethtool eth6 Settings  for  eth6: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) Link detected: yes [root@localhost pktgen]#

查看两个网口的中断号，并把eth5/eth6做亲和性绑定到特定的cpu（先关闭系统的irqbalance服务）：

1 2 3 4 5 6 7 8 9 10 11 12

[root@localhost ~]# cat /proc/interrupts | grep eth 16: 2586 540 1351588 4172554 IO-APIC-fasteoi Ensoniq AudioPCI, eth6 19: 5117 1949714 6098060 40 IO-APIC-fasteoi eth4, eth5 [root@localhost ~]# /etc/init.d/irqbalance stop Stopping irqbalance: [ OK ] [root@localhost ~]# echo 4 > /proc/irq/19/smp_affinity [root@localhost ~]# echo 8 > /proc/irq/16/smp_affinity [root@localhost ~]# cat /proc/irq/19/smp_affinity 04 [root@localhost ~]# cat /proc/irq/16/smp_affinity 08 [root@localhost ~]#

3，pktgen测试脚本如下（从文件：ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/pktgen.conf-2-2修改而来），可以看到相当于是把eth5和eth6拿根网线直连起来相互对发数据包：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

#! /bin/sh # FileName: pktgen-eth5-eth6.conf # modprobe pktgen function pgset() { local result echo $1 > $PGDEV result=`cat $PGDEV | fgrep  "Result: OK:" ` if  [  "$result"  =  ""  ]; then cat $PGDEV | fgrep Result: fi } function pg() { echo inject > $PGDEV cat $PGDEV } # Config Start Here ----------------------------------------------------------- # thread config # Each CPU has own thread. Two CPU exammple. We add eth1, eth2 respectivly. PGDEV=/proc/net/pktgen/kpktgend_2 echo  "Removing all devices" pgset  "rem_device_all" echo  "Adding eth5" pgset  "add_device eth5" PGDEV=/proc/net/pktgen/kpktgend_3 echo  "Removing all devices" pgset  "rem_device_all" echo  "Adding eth6" pgset  "add_device eth6" # device config # delay 0 means maximum speed. CLONE_SKB= "clone_skb 1000000" # NIC adds 4 bytes CRC PKT_SIZE= "pkt_size 60" # COUNT 0 means forever #COUNT="count 0" COUNT= "count 0" DELAY= "delay 0" PGDEV=/proc/net/pktgen/eth5 echo  "Configuring $PGDEV" pgset  "$COUNT" pgset  "$CLONE_SKB" pgset  "$PKT_SIZE" pgset  "$DELAY" pgset  "dst 192.168.1.96" pgset  "dst_mac 00:0C:29:97:9B:BE" PGDEV=/proc/net/pktgen/eth6 echo  "Configuring $PGDEV" pgset  "$COUNT" pgset  "$CLONE_SKB" pgset  "$PKT_SIZE" pgset  "$DELAY" pgset  "dst 192.168.1.95" pgset  "dst_mac 00:0C:29:97:9B:B4" # Time to run PGDEV=/proc/net/pktgen/pgctrl echo  "Running... ctrl^C to stop" pgset  "start" echo  "Done" # Result can be vieved in /proc/net/pktgen/eth[5,6]

4，执行上面那个pktgen测试脚本，看数据：

1 2 3 4 5 6 7 8

[root@localhost pktgen]# sh pktgen-eth5-eth6.conf Removing all devices Adding eth5 Removing all devices Adding eth6 Configuring /proc/net/pktgen/eth5 Configuring /proc/net/pktgen/eth6 Running... ctrl^C to stop

另开一个终端，执行mpstat命令查看cpu占用率，看网卡中断的处理是否正常：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

[root@localhost ~]# mpstat -P 2,3 1 Linux 2.6.38.8 (localhost.localdomain) 01/15/2012 _x86_64_ (4 CPU) 09:13:16 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 09:13:17 AM 2 0.00 0.00 86.87 0.00 0.00 13.13 0.00 0.00 0.00 09:13:17 AM 3 0.00 0.00 74.23 0.00 0.00 25.77 0.00 0.00 0.00 09:13:18 AM 2 0.00 0.00 87.63 0.00 0.00 12.37 0.00 0.00 0.00 09:13:18 AM 3 0.00 0.00 76.53 0.00 0.00 23.47 0.00 0.00 0.00 09:13:19 AM 2 0.00 0.00 85.71 0.00 0.00 14.29 0.00 0.00 0.00 09:13:19 AM 3 0.00 0.00 75.51 0.00 0.00 24.49 0.00 0.00 0.00 09:13:20 AM 2 0.00 0.00 86.60 0.00 0.00 13.40 0.00 0.00 0.00 09:13:20 AM 3 0.00 0.00 70.10 0.00 0.00 29.90 0.00 0.00 0.00 ^C [root@localhost ~]# cat /proc/interrupts | grep eth 16: 2586 540 588 6664664 IO-APIC-fasteoi Ensoniq AudioPCI, eth6 19: 5117 714 9729867 40 IO-APIC-fasteoi eth4, eth5 [root@localhost ~]#

执行一段时间后看统计数据：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

[root@localhost pktgen]# sh pktgen-eth5-eth6.conf Removing all devices Adding eth5 Removing all devices Adding eth6 Configuring /proc/net/pktgen/eth5 Configuring /proc/net/pktgen/eth6 Running... ctrl^C to stop ^C [root@localhost pktgen]# cat /proc/net/pktgen/eth5 Params: count 0 min_pkt_size: 60 max_pkt_size: 60 frags: 0 delay: 0 clone_skb: 1000000 ifname: eth5 flows: 0 flowlen: 0 queue_map_min: 0 queue_map_max: 0 dst_min: 192.168.1.96 dst_max: src_min: src_max: src_mac: 00:0c:29:97:9b:b4 dst_mac: 00:0c:29:97:9b:be udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9 src_mac_count: 0 dst_mac_count: 0 Flags: Current: pkts-sofar: 4405203 errors: 0 started: 2423250036us stopped: 2570673239us idle: 61432us seq_num: 4405204 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 cur_saddr: 0x5f01a8c0 cur_daddr: 0x6001a8c0 cur_udp_dst: 9 cur_udp_src: 9 cur_queue_map: 0 flows: 0 Result: OK: 147423202(c147361769+d61432) usec, 4405203 (60byte,0frags) 29881pps 14Mb/sec (14342880bps) errors: 0 [root@localhost pktgen]# cat /proc/net/pktgen/eth6 Params: count 0 min_pkt_size: 60 max_pkt_size: 60 frags: 0 delay: 0 clone_skb: 1000000 ifname: eth6 flows: 0 flowlen: 0 queue_map_min: 0 queue_map_max: 0 dst_min: 192.168.1.95 dst_max: src_min: src_max: src_mac: 00:0c:29:97:9b:be dst_mac: 00:0c:29:97:9b:b4 udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9 src_mac_count: 0 dst_mac_count: 0 Flags: Current: pkts-sofar: 2846965 errors: 0 started: 2423223982us stopped: 2570673243us idle: 45963us seq_num: 2846966 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 cur_saddr: 0x6001a8c0 cur_daddr: 0x5f01a8c0 cur_udp_dst: 9 cur_udp_src: 9 cur_queue_map: 0 flows: 0 Result: OK: 147449261(c147403297+d45963) usec, 2846965 (60byte,0frags) 19308pps 9Mb/sec (9267840bps) errors: 0 [root@localhost pktgen]#

看起来，这个测试结果也太差了，双向平均才2.5Wpps，后续再拿真实机器测试。

利用pktgen发包工具做性能测试（续）

2012年3月3日  lenky  没有评论  755 次浏览

最近在真实机器上测试了一下pktgen发包工具的性能，使用的pktgen来之：http://tslab.ssvl.kth.se/pktgen/，在Linux 2.6.37的内核上进行测试，原生pktgen源码为：http://tslab.ssvl.kth.se/pktgen/patches/net-next-v2.6.37/pktgen_rx.tgz，另外我对它做了一些改动，最主要的改动是可以将同一个网口按队列绑定到多个cpu上，收包也改成按网口分别统计。

基本软硬件环境为：Linux kernel 2.6.37 x86-64，ixgbe-3.8.21，4核心E5504（2.00GHz）的至强CPU，开启超线程后8个CPU线程，DDR3 1333 4G的内存，2个82599EB。

pktgen设置为“”clone_skb 0″、pkt_size 60″、”count 0″、”delay 0″”，两个万兆网口（eth0，eth1）设置为混杂模式并用光纤对接，加载ixgbe驱动后，网口自动按系统当前CPU线程数分为8个收发队列（eth0-TxRx-0、…、eth0-TxRx-7，eth1-TxRx-0、…、eth1-TxRx-7），将每一个收发队列绑定到一个CPU线程（cpu0处理eth0-TxRx-0和eth1-TxRx-0的收发、cpu1处理eth0-TxRx-1和eth1-TxRx-1的收发、…、cpu7处理eth0-TxRx-7和eth1-TxRx-7的收发），双向发收数据包（eth0和eth1同时发送和接收）。

测试结果：双向400Wpps，即eth0和eth1各自同时发送200Wpps，接收200Wpps，mpstat看到的cpu占用为 %sys/35.00，%soft/65.00，%idle/0.00。这是未做任何优化情况下的测试结果，额，貌似合符理论逻辑。

其它链接：
ftp://robur.slu.se/pub/Linux/bifrost/seminars/workshop-2011-03-31/
http://robur.slu.se/Linux/
http://caia.swin.edu.au/genius/tools/kute/