Oracle-RAC集群网络，IO性能测试

前言:

Oracle RAC集群对于网络，IO性能有一定的要求，特别是心跳网络，需要千兆以上，Oracle官方最佳实践建议配置万兆网络，而IO性能则是直接影响数据库的用户操作执行，因此，在进行RAC集群的部署时，我们需要对集群网络，IO性能进行测试，以确保性能指标满足集群运行要求。

本文接下来的内容将讲述Oracle RAC集群网络，IO性能常用的测试工具以及测试方法，希望对大家有所帮助。

测试工具:

网络:netpref

IO:orion,DBMS_RESOURCE_MANAGER.CALIBRATE_IO,fio

网络性能压测步骤(netpref)

安装编译netpref工具

tar -zxvf netperf-2.7.0.tar.gz
cd netperf-netperf-2.7.0/
./configure
make
make install

验证安装成功

[root@rac1 ~]# netperf -h
​
Usage: netperf [global options] -- [test options] 
​
Global options:
    -a send,recv      Set the local send,recv buffer alignment
    -A send,recv      Set the remote send,recv buffer alignment
    -B brandstr       Specify a string to be emitted with brief output
    -c [cpu_rate]     Report local CPU usage
    -C [cpu_rate]     Report remote CPU usage
    -d                Increase debugging output
    -D time,[units] * Display interim results at least every time interval
                      using units as the initial guess for units per second
                      A negative value for time will make heavy use of the
                      system's timestamping functionality
    -f G|M|K|g|m|k    Set the output units
    -F lfill[,rfill]* Pre-fill buffers with data from specified file
    -h                Display this text
    -H name|ip,fam *  Specify the target machine and/or local ip and family
    -i max,min        Specify the max and min number of iterations (15,1)
    -I lvl[,intvl]    Specify confidence level (95 or 99) (99) 
                      and confidence interval in percentage (10)
    -j                Keep additional timing statistics
    -l testlen        Specify test duration (>0 secs) (<0 bytes|trans)
    -L name|ip,fam *  Specify the local ip|name and address family
    -o send,recv      Set the local send,recv buffer offsets
    -O send,recv      Set the remote send,recv buffer offset
    -n numcpu         Set the number of processors for CPU util
    -N                Establish no control connection, do 'send' side only
    -p port,lport*    Specify netserver port number and/or local port
    -P 0|1            Don't/Do display test headers
    -r                Allow confidence to be hit on result only
    -s seconds        Wait seconds between test setup and test start
    -S                Set SO_KEEPALIVE on the data connection
    -t testname       Specify test to perform
    -T lcpu,rcpu      Request netperf/netserver be bound to local/remote cpu
    -v verbosity      Specify the verbosity level
    -W send,recv      Set the number of send,recv buffers
    -v level          Set the verbosity level (default 1, min 0)
    -V                Display the netperf version and exit
    -y local,remote   Set the socket priority
    -Y local,remote   Set the IP_TOS. Use hexadecimal.
    -Z passphrase     Set and pass to netserver a passphrase
​
For those options taking two parms, at least one must be specified;
specifying one value without a comma will set both parms to that
value, specifying a value with a leading comma will set just the second
parm, a value with a trailing comma will set just the first. To set
each parm to unique values, specify both and separate them with a
comma.
​
* For these options taking two parms, specifying one value with no comma
will only set the first parms and will leave the second at the default
value. To set the second value it must be preceded with a comma or be a
comma-separated pair. This is to retain previous netperf behaviour.

进行网络性能压测，netpref通过模拟服务端与客户端的TCP/UDP包网络发送，来测试主机之间的网络性能，这里测试的RAC为两节点的集群，所以节点一与节点二将分别作为服务段以及客户端进行测试

节点一RAC1作为服务端，启动netserver

[root@rac1 tmp]# netserver 
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC
#关闭netserver​,可以通过kill方式

节点二RAC2作为客户端，发送TCP/UDP进行测试

进行TCP协议测试

参数说明
#TCP_RR 在一个TCP连接中进行多次TCP request和response的交易过程，这种模式常常出现在数据库应用中
#TCP_STREAM: TCP批量传输，在测试过程中，netperf向netserver发送批量的TCP数据分组，以确定数据传输过程中的吞吐量
#-H 主机名IP
#-l 时长
[root@rac2 netperf-netperf-2.7.0]# netperf -t TCP_RR -H rac1-priv -l 60
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET : first burst 0
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   
​
16384  87380  1        1       60.00    1721.92   
16384  87380 
#Trans Rate per sec 每秒的交易1721/s
[root@rac2 netperf-netperf-2.7.0]# netperf -t TCP_STREAM -H rac1-priv -l 60
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  
​
 87380  16384  16384    60.00    2808.18   
 
 #Throughput 10^6bits/sec 网络吞吐量每秒2808Mbps/s

进行UDP协议测试

参数说明
#UDP_RR 进行多次UDP request和response的交易过程，这种模式常常出现在数据库应用中
#UDP_STREAM: UDP批量传输，在测试过程中，netperf向netserver发送批量的UDP数据分组，以确定数据传输过程中的吞吐量
#-H 主机名IP
#-l 时长
[root@rac2 netperf-netperf-2.7.0]# netperf -t UDP_RR -H rac1-priv  -l 60
MIGRATED UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET : first burst 0
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   
​
262144 262144 1        1       60.00    1727.69   
262144 262144
#Trans Rate per sec 每秒的交易1727/s
[root@rac2 netperf-netperf-2.7.0]# netperf -t UDP_STREAM -H rac1-priv -l 60
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec
​
262144   65507   60.00       44194      0     386.00
262144           60.00       44188            385.95
#Throughput 10^6bits/sec 网络吞吐量每秒385Mbps/s

总结:通过netpref的测试，我们可以看到TCP_RR的指标为每秒1721/s,TCP_STREAM每秒的吞吐量为2808Mbps/s,UDP_RR的指标为每秒1727/s,UDP_STREAM每秒的吞吐量为385Mbps/s,这大概就是千兆网络的性能

IO性能压测步骤(orion)

ORION (Oracle I/O Calibration Tool) 是校准用于 Oracle 数据库的存储系统 I/O 性能的独立工具。校准结果对于了解存储系统的性能有很大帮助，不仅可以找出影响 Oracle 数据库性能的问题，还能测量新数据库安装的大小。由于 ORION 是一个Oracle软件自带的独立工具，用户不需要创建和运行 Oracle 数据库

查看工具orion

[root@rac1 tmp]# /u01/app/oracle/product/11.2.0/db_1/bin/orion -help
ORION: ORacle IO Numbers -- Version 11.2.0.4.0
​
ORION runs IO performance tests that model Oracle RDBMS IO workloads.
It measures the performance of small (2-32K) IOs and large (128K+) IOs
at various load levels. 
​
Each Orion data point is a test for a specific mix of small and large 
IO loads sustained for a duration.  An Orion test consists of multiple 
data point tests.  These data point tests can be represented as a 
two-dimensional matrix.  Each column in the matrix represents data 
point tests with the same small IO load, but varying large IO loads.  
Each row represents data point tests with the same large IO load, but 
varying small IO loads.  An Orion test can be for a single point, a 
single row, a single column, or the whole matrix.
​
The 'run' parameter is the only mandatory parameter. Defaults
are indicated for all other parameters.  For additional information on
the user interface, see the Orion User Guide.
​
<testname> is the prefix used for all input and output filenames.  By
default, it is 'orion'.  It can be specified with the 'testname' 
parameter.
​
<testname>.lun should contain a carriage-return-separated list of LUNs.
​
The output files for a test run are prefixed by <testname>_<date> where
date is "yyyymmdd_hhmm".
​
The output files are: 
<testname>_<date>_summary.txt -  Summary of the input parameters, along with 
         the minimum small IO latency (in usecs), the maximum 
         MBPS, and the maximum IOPS observed. 
<testname>_<date>_mbps.csv - Performance results of large IOs in MBPS.
<testname>_<date>_iops.csv - Performance results of small IOs in IOPS.
<testname>_<date>_lat.csv - Latency of small IOs in microseconds.
<testname>_<date>_hist.csv - Histogram of IO latencies.
<testname>_<date>_trace.txt - Extended, unprocessed output.
​
WARNING: IF YOU ARE PERFORMING WRITE TESTS, BE PREPARED TO LOSE ANY DATA STORED
ON THE LUNS.
​
Mandatory parameters:
run      Type of workload to run (simple, normal, advanced, dss, oltp).
        simple   - Tests random small (8K) IOs at various loads,
                   then random large (1M) IOs at various loads.
        normal   - Tests combinations of random small (8K) IOs and 
                   random large (1M) IOs.
        advanced - Tests the workload specified by the user 
                   using optional parameters.
        dss      - Tests with random large (1M) IOs at increasing 
                   loads to determine the maximum throughput.
        oltp     - Tests with random small (8K) IOs at increasing 
                   loads to determine the maximum IOPS.
​
Optional parameters:
testname    Name of the test run.
​
num_disks    Number of disks (physical spindles).  This number is 
      used to gauge the range of loads that Orion should test 
      at.  Increasing this parameter results in Orion using 
      heavier IO loads.  Default is the number of LUNs in 
      <testname>.lun.
​
size_small    Size of small IOs in KB.  Default is 8.
​
size_large    Size of large IOs in KB.  Default is 1024.
​
type      Type of large IOs (rand, seq).  Default is rand.
        rand - Randomly distributed large IOs.
        seq  - Sequential streams of large IOs.
​
num_streamIO    Number of concurrent IOs per stream.  This parameter is only 
      used if -type is seq.  Default is 4.
​
simulate    Orion tests on a virtual LUN formed by combining the
      specified LUNs in one of these ways.  This parameter is 
      typically only used if -type is seq.  Default is concat.
        concat - A serial concatenation of the LUNs.  Each 
                 sequential stream issues IOs to only one LUN. 
        raid0  - A RAID-0 mapping across the LUNs.  Each 
                 sequential stream issues IOs across all LUNs, 
                 using RAID-0 striping.
​
write      Percentage of IOs that are writes (SEE WARNING ABOVE).
      Default is 0.
​
cache_size    Size in MBs of the array's cache.
      Unless this option is set to 0, Orion issues a number
      of unmeasured, random IOs before each large sequential
      data point.  These IOs fill up the storage array's cache
      (if any) with random data so that IOs from one
      data point do not result in cache hits for the next
      data point.  Read tests are preceded with junk reads
      and write tests are preceded with junk writes.  If
      specified, this 'cache warming' is performed until
      cache_size MBs of IO have been read or written.
      Default behavior is to issue 2 minutes of unmeasured random 
      IOs before each data point.
​
duration    Duration of each data point in seconds.  Default is 60.
​
num_small    Number of outstanding small IOs.  This parameter controls the 
      small IO load.  Only used if -matrix is point, col, or max.  
      No default.
​
num_large    This parameter controls the large IO load.
      For -type rand, number of outstanding large IOs.
      For -type seq, number of sequential IO streams.  Only used 
      if -matrix is point, row, or max.  No default.
​
matrix      An Orion test consists of multiple data point tests.  These data 
      point tests can be represented as a two-dimensional matrix. 
      Each column in the matrix represents data point tests with 
      the same small IO load, but varying large IO loads.  Each 
      row represents data point tests with the same large IO load, 
      but varying small IO loads.  An Orion test can be for a 
      single point, a single row, a single column, or the whole 
      matrix, depending on the matrix option setting below. 
      Default is basic.
        basic    - Test small IOs only, then large IOs only.
        detailed - Test entire matrix.
        point    - Test with num_small small IOs, num_large large 
                   IOs.
        col      - Test a varying large IO load with num_small 
                   small IOs.
        row      - Test a varying small IO load with num_large 
                   large IOs.
        max      - Test varying loads up to the num_small and 
                   num_large limits.
​
hugenotneeded    Tells Orion to allocate regular pages when huge
      pages are not available. With out this option, on systems
      that support hugepages, Orion errors out when it cannot
      allocate sufficient huge pages. -hugenotneeded is not set by
      default.
​
verbose      Prints tracing information to standard output if set.
      Not set by default.
datainput    Name of the input file for data to be written.
​
is_lgwr      if set to true will run test using VIP level QOS, assuming the underlying transport supports it.
​
​
Examples:
For a preliminary set of data
  -run simple 
For a basic set of data
  -run normal 
To evaluate storage for an OLTP database
  -run oltp 
To evaluate storage for a data warehouse
  -run dss 
To generate combinations of 32KB and 1MB reads to random locations
  -run advanced 
  -size_small 32 -size_large 1024 -type rand 
  -matrix detailed
To generate multiple sequential 1MB write streams, simulating RAID0 striping
  -run advanced 
  -simulate RAID0 -write 100 -type seq
  -matrix col -num_small 0
​

配置lun测试文件

#测试磁盘/dev/mapper/test1,/dev/mapper/test2,/dev/mapper/test3
[root@rac1 tmp]# cat tlun.lun 
/dev/mapper/test1
/dev/mapper/test2
/dev/mapper/test3

开始进行IO性能测试

#使用root用户
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH:/usr/local/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/rdbms/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
#测试参数
#模拟数据库随机小事务8K，读写占比7:3
#每个节点开启2个压测，共四个压测
/u01/app/oracle/product/11.2.0/db_1/bin/orion -run advanced -testname tlun  -num_disks 3 -size_small 8 -size_large 8 -type rand -write 30  -hugenotneeded
​
[root@rac1 tmp]# /u01/app/oracle/product/11.2.0/db_1/bin/orion -run advanced -testname tlun  -num_disks 3 -size_small 8 -size_large 8 -type rand -write 30  -hugenotneeded
ORION: ORacle IO Numbers -- Version 11.2.0.4.0
tlun_20220904_1717
Calibration will take approximately 23 minutes.
Using a large value for -cache_size may take longer.
​
​
Maximum Large MBPS=74.07 @ Small=0 and Large=6
​
Maximum Small IOPS=9583 @ Small=6 and Large=0
Small Read Latency: avg=623 us, min=85 us, max=10795 us, std dev=257 us @ Small=6 and Large=0
Small Write Latency: avg=627 us, min=94 us, max=10431 us, std dev=243 us @ Small=6 and Large=0
​
Minimum Small Latency=208 usecs @ Small=1 and Large=0
Small Read Latency: avg=232 us, min=78 us, max=6918 us, std dev=138 us @ Small=1 and Large=0
Small Write Latency: avg=151 us, min=88 us, max=4942 us, std dev=109 us @ Small=1 and Large=0
Small Read / Write Latency Histogram @ Small=1 and Large=0 
  Latency:    # of IOs (read)    # of IOs (write) 
        0 - 1    us:    0      0
        2 - 4    us:    0      0
        4 - 8    us:    0      0
        8 - 16    us:    0      0
       16 - 32    us:    0      0
       32 - 64    us:    0      0
       64 - 128    us:    42901      42021
      128 - 256    us:    98451      38491
      256 - 512    us:    57190      4334
      512 - 1024  us:    2413      549
     1024 - 2048  us:    522      173
     2048 - 4096  us:    126      37
     4096 - 8192  us:    15      3
     8192 - 16384  us:    0      0
    16384 - 32768  us:    0      0
    32768 - 65536  us:    0      0
    65536 - 131072  us:    0      0
   131072 - 262144  us:    0      0
   262144 - 524288  us:    0      0
   524288 - 1048576  us:    0      0
  1048576 - 2097152  us:    0      0
  2097152 - 4194304  us:    0      0
  4194304 - 8388608  us:    0      0
  8388608 - 16777216  us:    0      0
 16777216 - 33554432  us:    0      0
 33554432 - 67108864  us:    0      0
 67108864 - 134217728  us:    0      0
134217728 - 268435456  us:    0      0
---测试结果
Maximum Large MBPS=74.07
Maximum Small IOPS=9583

会在当前目录下生成测试的mbps,iops,latency的数据

[root@rac1 tmp]# ls -lrt tlun*
-rw-r--r-- 1 root root    54 Sep  4 17:13 tlun.lun
-rw-r--r-- 1 root root 20240 Sep  4 17:38 tlun_20220904_1717_trace.txt
-rw-r--r-- 1 root root   658 Sep  4 17:38 tlun_20220904_1717_mbps.csv
-rw-r--r-- 1 root root   758 Sep  4 17:38 tlun_20220904_1717_lat.csv
-rw-r--r-- 1 root root   734 Sep  4 17:38 tlun_20220904_1717_iops.csv
-rw-r--r-- 1 root root 20587 Sep  4 17:38 tlun_20220904_1717_hist.txt
-rw-r--r-- 1 root root  2151 Sep  4 17:38 tlun_20220904_1717_summary.txt

总结:测试的结果为最大IOPS:9583,最大吞吐量74Mbps/s

IO性能压测步骤(DBMS_RESOURCE_MANAGER.CALIBRATE_IO)

DBMS_RESOURCE_MANAGER.CALIBRATE_IO为Oracle 11g数据库中的PL/SQL存储过程，可以用于对当前数据库实例的I/O性能进行测试，使用简单

CALIBRATE_IO参数：

进行IO性能压测

set timing on
set time on
SET SERVEROUTPUT ON
DECLARE
  lat  INTEGER;
  iops INTEGER;
  mbps INTEGER;
BEGIN
  DBMS_RESOURCE_MANAGER.CALIBRATE_IO (num_physical_disks => 2,
                   max_iops      => iops,
                    max_mbps      => mbps,
                    actual_latency   => lat
);
  DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);
  DBMS_OUTPUT.PUT_LINE ('latency  = ' || lat);
  DBMS_OUTPUT.PUT_LINE ('max_mbps = ' || mbps);
end;
/

查看IO性能压测结果

SELECT TO_CHAR(start_time, 'DD-MON-YYY HH24:MI:SS') AS start_time,
   TO_CHAR(end_time, 'DD-MON-YYY HH24:MI:SS') AS end_time,
   max_iops,
   max_mbps,
   max_pmbps,
   latency,
   num_physical_disks AS disks
FROM  dba_rsrc_io_calibrate;18:03:16 SQL> 18:03:16 SQL> 18:03:16   2  18:03:16   3  18:03:16   4  18:03:16   5  18:03:16   6  18:03:16   7  18:03:16   8  
​
START_TIME         END_TIME     MAX_IOPS   MAX_MBPS  MAX_PMBPS    LATENCY      DISKS
---------------------- ---------------------- ---------- ---------- ---------- ---------- ----------
04-SEP-022 17:53:24    04-SEP-022 18:02:55     13070       1200     690         11     2

总结:测试的结果为最大IOPS:13070,最大吞吐量1200Mbps/s

IO性能压测步骤(FIO)

fio软件安装

tar xvf fio-2.17.tar.gz
cd fio-3.13
./configure 
make 
make install 

确认安装成功

[root@rac1 fio-2.17]# fio --help
fio-2.17
fio [options] [job options] <job file(s)>
  --debug=options  Enable debug logging. May be one/more of:
      process,file,io,mem,blktrace,verify,random,parse,
      diskutil,job,mutex,profile,time,net,rate,compress,
      steadystate,helperthread
  --parse-only    Parse options only, don't start any IO
  --output    Write output to file
  --bandwidth-log  Generate aggregate bandwidth logs
  --minimal    Minimal (terse) output
  --output-format=type  Output format (terse,json,json+,normal)
  --terse-version=type  Set terse version output format (default 3, or 2 or 4)
  --version    Print version info and exit
  --help    Print this page
  --cpuclock-test  Perform test/validation of CPU clock
  --crctest=type  Test speed of checksum functions
  --cmdhelp=cmd    Print command help, "all" for all of them
  --enghelp=engine  Print ioengine help, or list available ioengines
  --enghelp=engine,cmd  Print help for an ioengine cmd
  --showcmd    Turn a job file into command line options
  --eta=when    When ETA estimate should be printed
                May be "always", "never" or "auto"
  --eta-newline=time  Force a new line for every 'time' period passed
  --status-interval=t  Force full status dump every 't' period passed
  --readonly    Turn on safety read-only checks, preventing writes
  --section=name  Only run specified section in job file, multiple sections can be specified
  --alloc-size=kb  Set smalloc pool to this size in kb (def 16384)
  --warnings-fatal  Fio parser warnings are fatal
  --max-jobs=nr    Maximum number of threads/processes to support
  --server=args    Start a backend fio server
  --daemonize=pidfile  Background fio server, write pid to file
  --client=hostname  Talk to remote backend(s) fio server at hostname
  --remote-config=file  Tell fio server to load this local job file
  --idle-prof=option  Report cpu idleness on a system or percpu basis
      (option=system,percpu) or run unit work
      calibration only (option=calibrate)
  --inflate-log=log  Inflate and output compressed log
  --trigger-file=file  Execute trigger cmd when file exists
  --trigger-timeout=t  Execute trigger af this time
  --trigger=cmd    Set this command as local trigger
  --trigger-remote=cmd  Set this command as remote trigger
  --aux-path=path  Use this path for fio state generated files
​
Fio was written by Jens Axboe <jens.axboe@oracle.com>
                   Jens Axboe <jaxboe@fusionio.com>
                   Jens Axboe <axboe@fb.com>

使用参数说明

filename=/dev/mapper/test1 测试文件名称
direct=1 绕过操作系统的buffer，即进行O_DIRECT操作
iodepth  IO队列深度，在文件之上的保持的IO单元,默认对于每个文件来说是1，可以设置一个更大的值来提供并发度，iodepth大于1不会影响同步IO引擎(除非verify_async这个选项被设置)​
thread   io默认会使用fork()创建job，如果这个选项设置的话，fio将使用pthread_create来创建线程
rw=randwrite 测试随机写的I/O
rw=randrw    测试随机写和读的I/O
bs=8k 单次io的块文件大小为8k，仿照数据库的块大小
size=1g 本次的测试文件大小为1g，以每次8k的io进行测试。
numjobs=4 本次的测试线程为4.
runtime=300 测试时间为300秒，如果不写测试时间，则将1g文件按每8k写完结束
ioengine=psync io引擎使用pync方式
rwmixwrite=30 在混合读写的模式下，写占30%
rwmixread=70 在混合读写的模式下，读占70%
group_reporting 关于显示结果的，汇总每个进程的信息​
lockmem=1g 只使用1g内存进行测试​
zero_buffers 用0初始化系统buffer​
nrfiles=8 每个进程生成文件的数量

IO性能压测

---数据库场景主要是测试随机读写
---混合随机读写测试
[root@rac1 fio-2.17]# fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest  -ioscheduler=deadline
iotest: (g=0): rw=randrw, bs=8192B-8192B,8192B-8192B,8192B-8192B, ioengine=psync, iodepth=1
...
fio-2.17
Starting 4 threads
fio: os or kernel doesn't support IO scheduler switching
fio: os or kernel doesn't support IO scheduler switching
fio: os or kernel doesn't support IO scheduler switching
fio: os or kernel doesn't support IO scheduler switching
#最大读写吞吐量/IOPS
Jobs: 4 (f=4): [m(4)][100.0%][r=58.5MiB/s,w=25.4MiB/s][r=7486,w=3249 IOPS][eta 00m:00s]
iotest: (groupid=0, jobs=4): err= 0: pid=97958: Sun Sep  4 19:05:36 2022
   read: IOPS=6897, BW=53.1MiB/s (56.6MB/s)(2867MiB/53205msec)
    clat (usec): min=75, max=16751, avg=391.74, stdev=335.46
     lat (usec): min=75, max=16751, avg=392.36, stdev=335.54
    clat percentiles (usec):
     |  1.00th=[   80],  5.00th=[   92], 10.00th=[  108], 20.00th=[  147],
     | 30.00th=[  205], 40.00th=[  266], 50.00th=[  322], 60.00th=[  382],
     | 70.00th=[  454], 80.00th=[  548], 90.00th=[  724], 95.00th=[  924],
     | 99.00th=[ 1656], 99.50th=[ 2096], 99.90th=[ 3312], 99.95th=[ 3760],
     | 99.99th=[ 5984]
  write: IOPS=2956, BW=23.2MiB/s (24.3MB/s)(1229MiB/53205msec)
    clat (usec): min=82, max=16158, avg=412.63, stdev=334.53
     lat (usec): min=83, max=16158, avg=414.12, stdev=334.91
    clat percentiles (usec):
     |  1.00th=[   89],  5.00th=[   96], 10.00th=[  110], 20.00th=[  149],
     | 30.00th=[  219], 40.00th=[  290], 50.00th=[  354], 60.00th=[  418],
     | 70.00th=[  494], 80.00th=[  588], 90.00th=[  756], 95.00th=[  940],
     | 99.00th=[ 1592], 99.50th=[ 2064], 99.90th=[ 3312], 99.95th=[ 3824],
     | 99.99th=[ 6304]
    lat (usec) : 100=6.81%, 250=29.70%, 500=37.52%, 750=16.68%, 1000=5.18%
    lat (msec) : 2=3.53%, 4=0.53%, 10=0.04%, 20=0.01%
  cpu          : usr=1.31%, sys=19.35%, ctx=744673, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=366973,157315,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
​
Run status group 0 (all jobs):
   READ: bw=53.1MiB/s (56.6MB/s), 53.1MiB/s-53.1MiB/s (56.6MB/s-56.6MB/s), io=2867MiB (3006MB), run=53205-53205msec
  WRITE: bw=23.2MiB/s (24.3MB/s), 23.2MiB/s-23.2MiB/s (24.3MB/s-24.3MB/s), io=1229MiB (1289MB), run=53205-53205msec
​
---混合顺序读写测试：
[root@rac1 fio-2.17]# fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=rw -rwmixread=70 -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest  -ioscheduler=deadline
iotest: (g=0): rw=rw, bs=8192B-8192B,8192B-8192B,8192B-8192B, ioengine=psync, iodepth=1
...
fio-2.17
Starting 4 threads
fio: os or kernel doesn't support IO scheduler switching
fio: os or kernel doesn't support IO scheduler switching
fio: os or kernel doesn't support IO scheduler switching
fio: os or kernel doesn't support IO scheduler switching
Jobs: 4 (f=4): [M(4)][100.0%][r=48.5MiB/s,w=20.6MiB/s][r=6205,w=2627 IOPS][eta 00m:00s]
iotest: (groupid=0, jobs=4): err= 0: pid=99479: Sun Sep  4 19:09:41 2022
   read: IOPS=6943, BW=54.3MiB/s (56.9MB/s)(2867MiB/52851msec)
    clat (usec): min=73, max=13036, avg=392.24, stdev=346.79
     lat (usec): min=73, max=13036, avg=392.79, stdev=347.15
    clat percentiles (usec):
     |  1.00th=[   78],  5.00th=[   87], 10.00th=[  104], 20.00th=[  147],
     | 30.00th=[  199], 40.00th=[  262], 50.00th=[  318], 60.00th=[  378],
     | 70.00th=[  450], 80.00th=[  548], 90.00th=[  724], 95.00th=[  948],
     | 99.00th=[ 1752], 99.50th=[ 2192], 99.90th=[ 3408], 99.95th=[ 4128],
     | 99.99th=[ 5984]
  write: IOPS=2976, BW=23.3MiB/s (24.4MB/s)(1229MiB/52851msec)
    clat (usec): min=81, max=12001, avg=413.64, stdev=360.46
     lat (usec): min=81, max=12001, avg=414.78, stdev=360.97
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[   91], 10.00th=[  103], 20.00th=[  143],
     | 30.00th=[  207], 40.00th=[  278], 50.00th=[  350], 60.00th=[  410],
     | 70.00th=[  486], 80.00th=[  580], 90.00th=[  756], 95.00th=[  980],
     | 99.00th=[ 1784], 99.50th=[ 2256], 99.90th=[ 3472], 99.95th=[ 4192],
     | 99.99th=[ 6496]
    lat (usec) : 100=8.62%, 250=28.16%, 500=37.90%, 750=15.82%, 1000=5.00%
    lat (msec) : 2=3.81%, 4=0.63%, 10=0.06%, 20=0.01%
  cpu          : usr=0.80%, sys=19.45%, ctx=750369, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=366973,157315,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
​
Run status group 0 (all jobs):
   READ: bw=54.3MiB/s (56.9MB/s), 54.3MiB/s-54.3MiB/s (56.9MB/s-56.9MB/s), io=2867MiB (3006MB), run=52851-52851msec
  WRITE: bw=23.3MiB/s (24.4MB/s), 23.3MiB/s-23.3MiB/s (24.4MB/s-24.4MB/s), io=1229MiB (1289MB), run=52851-52851msec
​
---其他场景
---随机读测试：
fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest 
​
---随机写测试：
fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest
​
---顺序读测试：
fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1M -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest
​
---顺序写测试：
fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1M -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest

总结:

以上内容为Oracle RAC集群网络，IO性能常用的测试工具以及测试方法，大家可以根据自身的环境以及使用场景进行选择。