服务器性能测试 fio、sysbench、iperf3

lihongbao80

已于 2023-08-18 09:38:23 修改

阅读量1.2k

点赞数

分类专栏： linux 文章标签：服务器性能测试 sysbench fio iperf3

于 2023-06-28 13:45:49 首次发布

本文链接：https://blog.csdn.net/lihongbao80/article/details/131304869

版权

linux 专栏收录该内容

28 篇文章 3 订阅

订阅专栏

一、磁盘性能测试

安装fio磁盘性能测试工具

yum -y install epel-release
yum install -y fio
yum install libaio-devel

测试内容为顺序读、随机读、顺序写、随机写、随机读写

fio  -filename=/data/fiotest  -direct=1 -iodepth 1 -thread -rw=read -ioengine=libaio -bs=16k -size=50G -numjobs=30 -runtime=100 -group_reporting -name=readtest

fio  -filename=/data/fiotest  -direct=1 -iodepth 1 -thread -rw=randread -ioengine=libaio -bs=16k -size=50G -numjobs=30 -runtime=100 -group_reporting -name=randreadtest

fio -filename=/data/fiotest -direct=1 -iodepth 1 -thread -rw=write -ioengine=libaio -bs=16k -size=50G -numjobs=30 -runtime=100 -group_reporting -name=writetest

fio -filename=/data/fiotest -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=libaio -bs=16k -size=50G -numjobs=30 -runtime=100 -group_reporting -name=randwritetest

fio -filename=/data/fiotest -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=libaio -bs=16k -size=50G -numjobs=30 -runtime=100 -group_reporting -name=randrwtest

参数定义

filename=/tmp/fiotest 　支持文件系统或者裸设备，-filename=/dev/sda2或-filename=/dev/sdb 

direct=1 测试过程绕过机器自带的buffer，使测试结果更真实 

-thread：使用pthread_create创建线程，另一种是fork创建进程。进程的开销比线程要大，一般都采用thread测试。

rw=randwread 测试随机读的I/O 

rw=randwrite 测试随机写的I/O 

rw=randrw 测试随机混合写和读的I/O 

rw=read 测试顺序读的I/O 

rw=write 测试顺序写的I/O 

rw=rw 测试顺序混合写和读的I/O 

bs=4k 单次io的块文件大小为4k 

bsrange=512-2048 同上，提定数据块的大小范围 

size=50g 本次的测试文件大小为50g，以每次16k的io进行测试 

numjobs=30 本次的测试线程为30 

runtime=100 测试时间为100秒，如果不写则一直将50g文件分16k每次写完为止 

ioengine=libaio io引擎使用libaio方式

#
rwmixwrite=30 在混合读写的模式下，写占30% 

group_reporting 关于显示结果的，汇总每个进程的信息 

lockmem=1g 只使用1g内存进行测试 
zero_buffers 用0初始化系统
buffer nrfiles=8 每个进程生成文件的数量

结果分析

randreadtest: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 30 threads

randreadtest: (groupid=0, jobs=30): err= 0: pid=70058: Mon Feb 13 14:03:04 2023
   read: IOPS=394, BW=6304KiB/s (6455kB/s)(1848MiB/300101msec)
    slat (usec): min=5, max=2693, avg=34.48, stdev=14.56
    clat (usec): min=131, max=1014.7k, avg=76084.14, stdev=80707.24
     lat (usec): min=165, max=1014.7k, avg=76119.37, stdev=80707.24
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    9], 20.00th=[   16],
     | 30.00th=[   25], 40.00th=[   35], 50.00th=[   48], 60.00th=[   66],
     | 70.00th=[   90], 80.00th=[  124], 90.00th=[  182], 95.00th=[  241],
     | 99.00th=[  372], 99.50th=[  426], 99.90th=[  567], 99.95th=[  634],
     | 99.99th=[  802]
   bw (  KiB/s): min=   31, max=  640, per=3.34%, avg=210.53, stdev=86.17, samples=17962
   iops        : min=    1, max=   40, avg=13.10, stdev= 5.39, samples=17962
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.01%
  lat (msec)   : 2=0.05%, 4=1.56%, 10=9.94%, 20=13.73%, 50=25.94%
  lat (msec)   : 100=22.34%, 250=21.98%, 500=4.22%, 750=0.19%, 1000=0.01%
  cpu          : usr=0.02%, sys=0.07%, ctx=118608, majf=0, minf=3452
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=118242,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6304KiB/s (6455kB/s), 6304KiB/s-6304KiB/s (6455kB/s-6455kB/s), io=1848MiB (1937MB), run=300101-300101msec

Disk stats (read/write):
  sda: ios=118184/928, merge=0/2224, ticks=8989961/890698, in_queue=9821141, util=100.00%

指标主要看每秒 I/O 数（iops）和硬盘传输数据流的速度(BW)
在这里插入图片描述

二、CPU性能测试

安装sysbench性能测试工具

yum -y install epel-release
yum -y install sysbench

结果分析

pro70

sysbench cpu --cpu-max-prime=20000 --threads=8 --time=30 run
sysbench 1.0.17 (using system LuaJIT 2.0.4)

Running the test with following options:
Number of threads: 8
Initializing random number generator from current time


Prime numbers limit: 20000

Initializing worker threads...

Threads started!

CPU speed:
    events per second:  2787.23

General statistics:
    total time:                          30.0028s
    total number of events:              83630

Latency (ms):
         min:                                    2.70
         avg:                                    2.87
         max:                                   19.43
         95th percentile:                        3.36
         sum:                               239795.21

Threads fairness:
    events (avg/stddev):           10453.7500/47.54
    execution time (avg/stddev):   29.9744/0.01

Prime numbers limit: 20000，每个线程产生的素数上限均为2000
events per second: 2787.23，所有线程每秒完成了2787.23次event
total time: 30.0028s，共耗时30秒
total number of events: 83630，30秒内所有线程一共完成了83630次event
min: 2.70，完成1次event的最少耗时2.70秒
avg: 2.87，所有event的平均耗时2.87毫秒
max: 19.43，完成1次event的最多耗时19.43毫秒
95th percentile: 3.36， 95%次event在3.36秒毫秒内完成
sum: 239795.21，每个线程耗时30秒，8个线程叠加耗时就是240秒
events (avg/stddev): 10453.7500/47.54 // 平均每个线程完成10453次event，标准差为47
execution time (avg/stddev): 29.9744/0.01 // 每个线程平均耗时30秒，标准差为0

pro25

sysbench cpu --cpu-max-prime=20000 --threads=8 --time=30 run

sysbench 1.0.17 (using system LuaJIT 2.0.4)

Running the test with following options:
Number of threads: 8
Initializing random number generator from current time


Prime numbers limit: 20000

Initializing worker threads...

Threads started!

CPU speed:
    events per second:  3799.96

General statistics:
    total time:                          30.0027s
    total number of events:              114027

Latency (ms):
         min:                                    2.03
         avg:                                    2.10
         max:                                    7.76
         95th percentile:                        2.39
         sum:                               239966.24

Threads fairness:
    events (avg/stddev):           14253.3750/22.81
    execution time (avg/stddev):   29.9958/0.00

Prime numbers limit: 20000，每个线程产生的素数上限均为2000
events per second: 3799.96 ，所有线程每秒完成了3799.96次event
total time: 30.0027s，共耗时30秒
total number of events: 114027，30秒内所有线程一共完成了114027次event
min: 2.03，完成1次event的最少耗时2.03秒
avg: 2.10，所有event的平均耗时2.10毫秒
max: 7.76，完成1次event的最多耗时7.76毫秒
95th percentile: 2.39， 95%次event在2.39秒毫秒内完成
sum: 239966.24，每个线程耗时30秒，8个线程叠加耗时就是240秒
events (avg/stddev): 14253.3750/22.81 // 平均每个线程完成14253.3750/22.81次event，标准差为23
execution time (avg/stddev): 29.9958/0.00 // 每个线程平均耗时30秒，标准差为0

三、内存性能测试

结果分析

pro70

sysbench memory --threads=12 --memory-block-size=8K --memory-total-size=100G --memory-access-mode=seq run
sysbench 1.0.17 (using system LuaJIT 2.0.4)

Running the test with following options:
Number of threads: 12
Initializing random number generator from current time


Running memory speed test with the following options:
  block size: 8KiB
  total size: 102400MiB
  operation: write
  scope: global

Initializing worker threads...

Threads started!

Total operations: 10112320 (1011060.89 per second)

79002.50 MiB transferred (7898.91 MiB/sec)


General statistics:
    total time:                          10.0001s
    total number of events:              10112320

Latency (ms):
         min:                                    0.00
         avg:                                    0.01
         max:                                   28.03
         95th percentile:                        0.01
         sum:                               116284.49

Threads fairness:
    events (avg/stddev):           842693.3333/45270.47
    execution time (avg/stddev):   9.6904/0.04




total number of events:  13107200    读/写一个内存块的events数
min:  0.00  最小耗时 这里等于0是因为内存块太小,返回的结果数值精度不够
sum: 62961.90    所有线程总耗时62962ms
events (avg/stddev):  1638400.0000/0.00   平均每线程完成events数1638400/标准差0
execution time (avg/stddev):   7.8702/1.15      平均每线程执行时间8s 标准差1s

pro25

sysbench memory --threads=12 --memory-block-size=8K --memory-total-size=100G --memory-access-mode=seq run
sysbench 1.0.17 (using system LuaJIT 2.0.4)

Running the test with following options:
Number of threads: 12
Initializing random number generator from current time


Running memory speed test with the following options:
  block size: 8KiB
  total size: 102400MiB
  operation: write
  scope: global

Initializing worker threads...

Threads started!

Total operations: 7038742 (703511.93 per second)

54990.17 MiB transferred (5496.19 MiB/sec)


General statistics:
    total time:                          10.0002s
    total number of events:              7038742

Latency (ms):
         min:                                    0.00
         avg:                                    0.02
         max:                                    0.49
         95th percentile:                        0.02
         sum:                               118420.01

Threads fairness:
    events (avg/stddev):           586561.8333/6249.45
    execution time (avg/stddev):   9.8683/0.00

total number of events:  7038742   读/写一个内存块的events数
min:  0.00  最小耗时 这里等于0是因为内存块太小,返回的结果数值精度不够
sum: 118420.01    所有线程总耗时118420ms
events (avg/stddev):  586561.8333/6249.45   平均每线程完成events数586562/标准差6249
execution time (avg/stddev):   9.8683/0.00      平均每线程执行时间10s 标准差0s

四、网络性能测试

测试iperf3工具

yum install -y sysstat wget tar automake make gcc
wget https://downloads.es.net/pub/iperf/iperf-3.9.tar.gz
tar -zxvf iperf-3.9.tar.gz 
cd iperf-3.9
./configure && make && make install

测试结果分析

要使用iperf，首先要启用一个服务端，这里指定服务端的IP地址为pro70，在此服务器上运行“iperf3 -s -i 2”即可开启iperf的服务器模式。在默认情况下，iperf3将在服务端打开一个5201监听端口，此时就可以在pro25服务器作为客户端执行iperf功能测试了。

pro70

[root@pro70 ~]# iperf3 -s -i 2 
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

1.测试TCP吞吐量

pro25

[root@pro25 ~]# iperf3 -c 192.168.16.70 -F /data/fiotest -i 5 -t 180
Connecting to host 192.168.16.70, port 5201
[  5] local 192.168.16.25 port 51856 connected to 192.168.16.70 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-5.00   sec  4.76 GBytes  8.17 Gbits/sec  670   1.18 MBytes       
[  5]   5.00-10.00  sec  4.80 GBytes  8.25 Gbits/sec  237    854 KBytes       
[  5]  10.00-15.00  sec  4.78 GBytes  8.21 Gbits/sec  1172    604 KBytes       
[  5]  15.00-20.01  sec  2.21 GBytes  3.79 Gbits/sec   28   1014 KBytes       
[  5]  20.01-25.00  sec  1.10 GBytes  1.90 Gbits/sec   42    788 KBytes       
[  5]  25.00-30.00  sec  1.06 GBytes  1.83 Gbits/sec   71    274 KBytes       
[  5]  30.00-35.00  sec  1.08 GBytes  1.85 Gbits/sec   32    389 KBytes       
[  5]  35.00-40.00  sec  1.09 GBytes  1.86 Gbits/sec   20    365 KBytes       
[  5]  40.00-45.00  sec  1.10 GBytes  1.88 Gbits/sec   98    304 KBytes       
[  5]  45.00-50.00  sec  1.08 GBytes  1.85 Gbits/sec   22    393 KBytes       
[  5]  50.00-55.00  sec  1.10 GBytes  1.88 Gbits/sec    1    406 KBytes       
[  5]  55.00-60.00  sec  1.06 GBytes  1.82 Gbits/sec   43    337 KBytes       
[  5]  60.00-65.00  sec  1.10 GBytes  1.88 Gbits/sec    0    397 KBytes       
[  5]  65.00-70.00  sec  1.09 GBytes  1.88 Gbits/sec    0    403 KBytes       
[  5]  70.00-75.00  sec  1.10 GBytes  1.89 Gbits/sec    0    419 KBytes       
[  5]  75.00-80.00  sec  1.07 GBytes  1.84 Gbits/sec    0    427 KBytes       
[  5]  80.00-85.00  sec  1.09 GBytes  1.88 Gbits/sec    0    434 KBytes       
[  5]  85.00-90.00  sec  1.09 GBytes  1.88 Gbits/sec    0    443 KBytes       
[  5]  90.00-95.00  sec  1.09 GBytes  1.87 Gbits/sec    0    448 KBytes       
[  5]  95.00-100.02 sec  1.06 GBytes  1.81 Gbits/sec    0    457 KBytes       
[  5] 100.02-105.00 sec  1.09 GBytes  1.88 Gbits/sec    0    465 KBytes       
[  5] 105.00-110.00 sec  1.06 GBytes  1.82 Gbits/sec    0    475 KBytes       
[  5] 110.00-115.00 sec  1.09 GBytes  1.87 Gbits/sec    0    496 KBytes       
[  5] 115.00-120.01 sec  1.09 GBytes  1.88 Gbits/sec    0    663 KBytes       
[  5] 120.01-125.00 sec  1.09 GBytes  1.88 Gbits/sec    0    663 KBytes       
[  5] 125.00-130.00 sec  1.09 GBytes  1.87 Gbits/sec    0   1003 KBytes       
[  5] 130.00-135.00 sec  1.06 GBytes  1.82 Gbits/sec    0   1003 KBytes       
[  5] 135.00-140.00 sec  1.08 GBytes  1.86 Gbits/sec    0   1003 KBytes       
[  5] 140.00-145.00 sec  1.10 GBytes  1.89 Gbits/sec    0   1003 KBytes       
[  5] 145.00-150.00 sec  1.09 GBytes  1.87 Gbits/sec    0   1.42 MBytes       
[  5] 150.00-155.00 sec  1.09 GBytes  1.87 Gbits/sec    0   1.42 MBytes       
[  5] 155.00-160.00 sec  1.09 GBytes  1.88 Gbits/sec    0   1.42 MBytes       
[  5] 160.00-165.00 sec  1.09 GBytes  1.87 Gbits/sec    0   1.42 MBytes       
[  5] 165.00-170.00 sec  1.04 GBytes  1.78 Gbits/sec   33   1.01 MBytes       
[  5] 170.00-174.36 sec   982 MBytes  1.89 Gbits/sec    0   1.01 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-174.36 sec  50.0 GBytes  2.46 Gbits/sec  2469             sender
        Sent 50.0 GByte / 50.0 GByte (100%) of /data/fiotest
[  5]   0.00-174.36 sec  50.0 GBytes  2.46 Gbits/sec                  receiver

iperf Done.

在这此测试中我们通过“-F”参数指定了一个50G的fiotest文件作为iperf要传输的数据,需要同时指定一个“-t”参数来设置要测试传输的时间，这个时间尽量设置长一些，因为在默认传输时间10秒内，这个文件可能还没有传完。

通过结果我们可以看到传输速率平均在2.46 Gbits/sec

2 . 测试UDP丢包和延迟

iperf也可以用于UDP数据包吞吐量、丢包率和延迟指标，但是由于UDP协议是一个非面向连接的轻量级传输协议，并且不提供可靠的数据传输服务，因此对UDP应用的关注点不是传输数据有多快，而是它的丢包率和延时指标。通过iperf的“-u”参数即可测试UDP应用的传输性能

pro70

[root@pro25 ~]# iperf3 -c 192.168.16.70 -u -b 1G -f M -i 3
Connecting to host 192.168.16.70, port 5201
[  5] local 192.168.16.25 port 47184 connected to 192.168.16.70 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-3.00   sec   358 MBytes   119 MBytes/sec  258918  
[  5]   3.00-6.00   sec   358 MBytes   119 MBytes/sec  258955  
[  5]   6.00-9.00   sec   356 MBytes   119 MBytes/sec  257490  
[  5]   9.00-10.00  sec   121 MBytes   121 MBytes/sec  87860  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes   119 MBytes/sec  0.000 ms  0/863223 (0%)  sender
[  5]   0.00-10.00  sec  1.16 GBytes   118 MBytes/sec  0.008 ms  5513/863223 (0.64%)  receiver

iperf Done.