Borker Load 导入导出性能测试

测试环境

机器配置

FIO测试

starrocks: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
starrocks: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6232KiB/s][r=0,w=779 IOPS][eta 00m:00s]
starrocks: (groupid=0, jobs=1): err= 0: pid=18050: Thu Feb 17 15:01:01 2022
  write: IOPS=789, BW=6314KiB/s (6466kB/s)(370MiB/60001msec)
    clat (usec): min=6, max=2415, avg=13.37, stdev=51.60
     lat (usec): min=6, max=2416, avg=13.56, stdev=51.61
    clat percentiles (usec):
     |  1.00th=[    8],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[    9], 40.00th=[   10], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   13], 90.00th=[   15], 95.00th=[   18],
     | 99.00th=[   24], 99.50th=[   27], 99.90th=[  832], 99.95th=[  963],
     | 99.99th=[ 2040]
   bw (  KiB/s): min= 3696, max= 6528, per=100.00%, avg=6313.89, stdev=278.36, samples=120
   iops        : min=  462, max=  816, avg=789.25, stdev=34.80, samples=120
  lat (usec)   : 10=55.38%, 20=41.83%, 50=2.47%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.24%
  lat (msec)   : 2=0.03%, 4=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=845, max=68060, avg=1252.20, stdev=507.46
    sync percentiles (usec):
     |  1.00th=[ 1156],  5.00th=[ 1156], 10.00th=[ 1156], 20.00th=[ 1172],
     | 30.00th=[ 1172], 40.00th=[ 1172], 50.00th=[ 1188], 60.00th=[ 1188],
     | 70.00th=[ 1205], 80.00th=[ 1270], 90.00th=[ 1336], 95.00th=[ 1500],
     | 99.00th=[ 2343], 99.50th=[ 2540], 99.90th=[ 4178], 99.95th=[ 5276],
     | 99.99th=[21890]
  cpu          : usr=0.17%, sys=2.29%, ctx=95588, majf=0, minf=36
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,47356,0,0 short=47356,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=6314KiB/s (6466kB/s), 6314KiB/s-6314KiB/s (6466kB/s-6466kB/s), io=370MiB (388MB), run=60001-60001msec

Disk stats (read/write):
  vdb: ios=142/143978, merge=0/99169, ticks=316/63247, in_queue=42906, util=62.55%

StarRocks 集群信息

HDFS 集群信息

测试模型

测试采用 TPCH 50G 测试集

测试模型信息

表结构及数据

表结构

drop table if exists customer;
CREATE TABLE customer (
    c_custkey     int NOT NULL,
    c_name        VARCHAR(25) NOT NULL,
    c_address     VARCHAR(40) NOT NULL,
    c_nationkey   int NOT NULL,
    c_phone       VARCHAR(15) NOT NULL,
    c_acctbal     decimal(15, 2)   NOT NULL,
    c_mktsegment  VARCHAR(10) NOT NULL,
    c_comment     VARCHAR(117) NOT NULL
)ENGINE=OLAP
DUPLICATE KEY(`c_custkey`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 24
PROPERTIES (
    "replication_num" = "3",
    "in_memory" = "false",
    "storage_format" = "DEFAULT"
);


drop table if exists lineitem;
CREATE TABLE lineitem ( 
    l_shipdate    DATE ,
    l_orderkey    int ,
    l_linenumber  int ,
    l_partkey     int ,
    l_suppkey     int ,
    l_quantity    decimal(15, 2) ,
    l_extendedprice  decimal(15, 2) ,
    l_discount    decimal(15, 2) ,
    l_tax         decimal(15, 2) ,
    l_returnflag  VARCHAR(1) ,
    l_linestatus  VARCHAR(1) ,
    l_commitdate  DATE ,
    l_receiptdate DATE ,
    l_shipinstruct VARCHAR(25) ,
    l_shipmode     VARCHAR(10) ,
    l_comment      VARCHAR(44) 
)ENGINE=OLAP
DUPLICATE KEY(`l_shipdate`, `l_orderkey`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`l_orderkey`) BUCKETS 96
PROPERTIES (
    "replication_num" = "3",
    "in_memory" = "false",
    "storage_format" = "DEFAULT"
);


drop table if exists nation;
CREATE TABLE `nation` (
  `n_nationkey` int(11) NOT NULL,
  `n_name`      varchar(25) NOT NULL,
  `n_regionkey` int(11) NOT NULL,
  `n_comment`   varchar(152) NULL
) ENGINE=OLAP
DUPLICATE KEY(`N_NATIONKEY`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`N_NATIONKEY`) BUCKETS 1
PROPERTIES (
    "replication_num" = "3",
    "in_memory" = "false",
    "storage_format" = "DEFAULT"
);


drop table if exists orders;
CREATE TABLE orders  (
    o_orderkey       int ,
    o_orderdate      DATE ,
    o_custkey        int ,
    o_orderstatus    VARCHAR(1) ,
    o_totalprice     decimal(15, 2) ,
    o_orderpriority  VARCHAR(15) ,
    o_clerk          VARCHAR(15) ,
    o_shippriority   int ,
    o_comment        VARCHAR(79) 
)ENGINE=OLAP
DUPLICATE KEY(`o_orderkey`, `o_orderdate`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`o_orderkey`) BUCKETS 96
PROPERTIES (
    "replication_num" = "3",