使用 Iperf 测试软路由性能

目标

有 openwrt、pfsense、ikuai、routeros 等4个软路由，希望通过iperf工具对他们的纯包转发速率、带宽限制/IP、连接数限制/IP 效果进行比对。

Iperf

Iperf 是网络测速工具。官网地址： https://github.com/esnet/iperf

官网

https://github.com/esnet/iperf

版本

有iperf 和iperf3 两个版本，因为iperf3 限制客户端最大连接数128，所以这里选择使用iperf。

软件安装

sudo apt install iperf

命令行帮助信息

Usage: iperf [-s|-c host] [options]
       iperf [-h|--help] [-v|--version]

Client/Server:
  -b, --bandwidth #[kmgKMG | pps]  bandwidth to send at in bits/sec or packets per second
  -e, --enhancedreports    use enhanced reporting giving more tcp/udp and traffic information
  -f, --format    [kmgKMG]   format to report: Kbits, Mbits, KBytes, MBytes
  -i, --interval  #        seconds between periodic bandwidth reports
  -l, --len       #[kmKM]    length of buffer in bytes to read or write (Defaults: TCP=128K, v4 UDP=1470, v6 UDP=1450)
  -m, --print_mss          print TCP maximum segment size (MTU - TCP/IP header)
  -o, --output    <filename> output the report or error message to this specified file
  -p, --port      #        server port to listen on/connect to
  -u, --udp                use UDP rather than TCP
      --udp-counters-64bit use 64 bit sequence numbers with UDP
  -w, --window    #[KM]    TCP window size (socket buffer size)
  -z, --realtime           request realtime scheduler
  -B, --bind <host>[:<port>][%<dev>] bind to <host>, ip addr (including multicast address) and optional port and device
  -C, --compatibility      for use with older versions does not sent extra msgs
  -M, --mss       #        set TCP maximum segment size (MTU - 40 bytes)
  -N, --nodelay            set TCP no delay, disabling Nagle's Algorithm
  -S, --tos       #        set the socket's IP_TOS (byte) field

Server specific:
  -s, --server             run in server mode
  -t, --time      #        time in seconds to listen for new connections as well as to receive traffic (default not set)
      --udp-histogram #,#  enable UDP latency histogram(s) with bin width and count, e.g. 1,1000=1(ms),1000(bins)
  -B, --bind <ip>[%<dev>]  bind to multicast address and optional device
  -H, --ssm-host <ip>      set the SSM source, use with -B for (S,G) 
  -U, --single_udp         run in single threaded UDP mode
  -D, --daemon             run the server as a daemon
  -V, --ipv6_domain        Enable IPv6 reception by setting the domain and socket to AF_INET6 (Can receive on both IPv4 and IPv6)

Client specific:
  -c, --client    <host>   run in client mode, connecting to <host>
  -d, --dualtest           Do a bidirectional test simultaneously
      --ipg                set the the interpacket gap (milliseconds) for packets within an isochronous frame
      --isochronous <frames-per-second>:<mean>,<stddev> send traffic in bursts (frames - emulate video traffic)
  -n, --num       #[kmgKMG]    number of bytes to transmit (instead of -t)
  -r, --tradeoff           Do a bidirectional test individually
  -t, --time      #        time in seconds to transmit for (default 10 secs)
  -B, --bind [<ip> | <ip:port>] bind ip (and optional port) from which to source traffic
  -F, --fileinput <name>   input the data to be transmitted from a file
  -I, --stdin              input the data to be transmitted from stdin
  -L, --listenport #       port to receive bidirectional tests back on
  -P, --parallel  #        number of parallel client threads to run
  -R, --reverse            reverse the test (client receives, server sends)
  -T, --ttl       #        time-to-live, for multicast (default 1)
  -V, --ipv6_domain        Set the domain to IPv6 (send packets over IPv6)
  -X, --peer-detect        perform server version detection and version exchange
  -Z, --linux-congestion <algo>  set TCP congestion control algorithm (Linux only)

Miscellaneous:
  -x, --reportexclude [CDMSV]   exclude C(connection) D(data) M(multicast) S(settings) V(server) reports
  -y, --reportstyle C      report as a Comma-Separated Values
  -h, --help               print this message and quit
  -v, --version            print version information and quit

[kmgKMG] Indicates options that support a k,m,g,K,M or G suffix
Lowercase format characters are 10^3 based and uppercase are 2^n based
(e.g. 1k = 1000, 1K = 1024, 1m = 1,000,000 and 1M = 1,048,576)

The TCP window size option can be set by the environment variable
TCP_WINDOW_SIZE. Most other options can be set by an environment variable
IPERF_<long option name>, such as IPERF_BANDWIDTH.

Source at <http://sourceforge.net/projects/iperf2/>
Report bugs to <iperf-users@lists.sourceforge.net>

测试方案

因为我们的目标是比较各软路由的性能，所以只需要关注端口、连接数、数据发送的大小、数据发送的时间等参数即可，TCP\UDP的其他参数，Linux内核的参数等均不在此次测试的范围内。

拓扑

为排除其他干扰，在单独的交换机上搭建以下服务。

Ubuntu-server（iperf 服务端） --WAN–软路由–LAN-- Ubuntu-server（iperf 客户端）

打开ulimit

对于linux来讲，打开连接就是打开文件，此时需要将测试用户（root)的文件打开数限制打开，服务器和客户端都要打开。

ulimit -n 10000

服务器端命令

iperf  -s  
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------

客户端命令

# 单连接连续测试120秒
iperf -c x.x.x.x  -t 120 
# 100 连接连续测试120秒
iperf -c x.x.x.x  -t 120 -P 100
# 1000 连接连续测试120秒
iperf -c x.x.x.x  -t 120 -P 1000
# 2000 连接连续测试120秒
iperf -c x.x.x.x  -t 120 -P 2000
# 根据服务器性能将连接数最高加到1W

裸机包转发速率测试

以上软路由主要依赖内核的包过滤功能进行数据转发，网络策略越复杂，对性能的影响越大。我们首先清空4种路由器的网络策略，使用空白的策略进行包转发测试。

限制网络速度/IP 效果测试

限制 连接数/IP 效果测试