Doris集群环境搭建

本文记录按照官网文档进行操作过程

1、下载预编译包

（1）百度Doris官方网址
2021年，百度网址doris.baidu.com/doc已经调整为了
https://cloud.baidu.com/doc/PALO/s/Wksis5irl

请注意官方提示：

• 适用于CentOS 7.3
• 依赖Oracle JDK 1.8 版本

（2）确定CPU是否支持avx2指令

检查如下，我使用的是虚拟机，暂不支持avx2指令，因此需要下载对应的no-avx2版本的预编译

[root@node3 ~]# cat /proc/cpuinfo | grep flags
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
[root@node3 ~]# cat /proc/cpuinfo | grep avx2
[root@node3 ~]# 

（3）下载
如上图，单击0.14.13-no-avx2后面的Link，进行下载
https://palo-cloud-repo-bd.bd.bcebos.com/baidu-doris-release/PALO-0.14.13-release-no-avx2-binary.tar.gz

2、FE基本配置

（1）将软件包上传到服务器
此处我上传到了自己创建的/app目录下

（2）加压并重命名

[root@node3 app]# tar -zxvf PALO-0.14.13-release-no-avx2-binary.tar.gz
[root@node3 app]# mv PALO-0.14.13-release-no-avx2-binary doris-0.14.13
[root@node3 app]# cd doris-0.14.13/
[root@node3 doris-0.14.13]# ls
apache_hdfs_broker  auditloader.zip  be  doris-flink-1.0-SNAPSHOT.jar  doris-spark-1.0.0-SNAPSHOT.jar  fe  udf
[root@node3 doris-0.14.13]#

（3）创建FE节点元数据目录

[root@node3 doris-0.14.13]# cd fe
[root@node3 fe]# mkdir doris-meta
[root@node3 fe]# 

（4）配置参数

[root@node3 fe]# vi conf/fe.conf

修改JAVA_OPTS参数，默认内存是4G，根据具体情况修改。生产环境建议8G以上，我这里是测试环境，修改为了1G（1024M）

JAVA_OPTS="-Xmx1024m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

配置priority_networks

priority_networks = 10.17.12.160/24

3、BE基本配置

（1）配置环境参数/etc/security/limits.conf
如果这一步已经配置过了，可以跳过。

[root@node1 be]# vi /etc/security/limits.conf

在文件末尾添加如下内容

# End of file

* hard nofile 65536
* soft nofile 65536

（2）创建数据目录

[root@node3 fe]# cd ../be
[root@node3 be]# mkdir storage
[root@node3 be]# 

（3）修改配置文件

[root@node3 be]# vi conf/be.conf

priority_networks = 10.17.12.160/24

4、配置集群节点

（1）复制软件包到集群节点

[root@node3 app]# scp -r doris-0.14.13/ node1:/app
[root@node3 app]# scp -r doris-0.14.13/ node2:/app

（2）修改FE配置
先修改node1

[root@node1 ~]# cd /app/doris-0.14.13/
[root@node1 doris-0.14.13]# vi fe/conf/fe.conf 
[root@node1 doris-0.14.13]#

priority_networks = 10.17.12.158/24

在修改node2

[root@node2 ~]# cd /app/doris-0.14.13/
[root@node2 doris-0.14.13]# vi fe/conf/fe.conf 
[root@node2 doris-0.14.13]#

priority_networks = 10.17.12.159/24

（3）修改BE配置

[root@node1 doris-0.14.13]# vi be/conf/be.conf

[root@node2 doris-0.14.13]# vi be/conf/be.conf

修改内容：
node1:

priority_networks = 10.17.12.158/24

node2

priority_networks = 10.17.12.159/24

注意：如果是第一次配置了/etc/security/limits.conf，请重启机器。

5、 启动FE 集群

（1）启动第一个FE节点

[root@node1 app]# cd doris-0.14.13/fe
[root@node1 fe]# sh bin/start_fe.sh --daemon
[root@node1 fe]# jps
10400 PaloFe
12744 Worker
12249 Master
10475 Jps
[root@node1 fe]#

（2）登录客户端
参考https://blog.csdn.net/chengyuqiang/article/details/119974926按照mysql-client客户端

[root@node1 fe]# mysql -h10.17.12.158 -P9030 -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.1.0

Copyright (c) 2000, 2021, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

（3）添加FE节点

mysql> ALTER SYSTEM ADD FOLLOWER "node2:9010";
Query OK, 0 rows affected (0.04 sec)

mysql> ALTER SYSTEM ADD FOLLOWER "node3:9010";
Query OK, 0 rows affected (0.01 sec)

mysql>

（4）查看FE集群状态

mysql> show frontends \G
*************************** 1. row ***************************
             Name: 10.17.12.158_9010_1630292462734
               IP: 10.17.12.158
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: true
        ClusterId: 1583270145
             Join: true
            Alive: true
ReplayedJournalId: 219
    LastHeartbeat: 2021-08-30 11:13:04
         IsHelper: true
           ErrMsg: 
          Version: 0.14.13-release-Unknown
*************************** 2. row ***************************
             Name: 10.17.12.160_9010_1630293081245
               IP: 10.17.12.160
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 0
          RpcPort: 0
             Role: FOLLOWER
         IsMaster: false
        ClusterId: 1583270145
             Join: false
            Alive: false
ReplayedJournalId: 0
    LastHeartbeat: NULL
         IsHelper: true
           ErrMsg: got exception
          Version: NULL
*************************** 3. row ***************************
             Name: 10.17.12.159_9010_1630293077685
               IP: 10.17.12.159
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 0
          RpcPort: 0
             Role: FOLLOWER
         IsMaster: false
        ClusterId: 1583270145
             Join: false
            Alive: false
ReplayedJournalId: 0
    LastHeartbeat: NULL
         IsHelper: true
           ErrMsg: got exception
          Version: NULL
3 rows in set (0.06 sec)

mysql> 

（5）启动其他两个FE节点

[root@node2 fe]# ./bin/start_fe.sh --helper node1:9010 --daemon
[root@node2 fe]# jps
6144 Worker
18516 PaloFe
18568 Jps
[root@node2 fe]#

[root@node3 fe]# ./bin/start_fe.sh --helper node1:9010 --daemon
[root@node3 fe]# jps
5971 PaloFe
1940 Worker
6025 Jps
[root@node3 fe]# 

mysql> show frontends \G
*************************** 1. row ***************************
             Name: 10.17.12.158_9010_1630292462734
               IP: 10.17.12.158
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: true
        ClusterId: 1583270145
             Join: true
            Alive: true
ReplayedJournalId: 286
    LastHeartbeat: 2021-08-30 11:16:50
         IsHelper: true
           ErrMsg: 
          Version: 0.14.13-release-Unknown
*************************** 2. row ***************************
             Name: 10.17.12.160_9010_1630293081245
               IP: 10.17.12.160
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: false
        ClusterId: 1583270145
             Join: true
            Alive: true
ReplayedJournalId: 285
    LastHeartbeat: 2021-08-30 11:16:50
         IsHelper: true
           ErrMsg: 
          Version: 0.14.13-release-Unknown
*************************** 3. row ***************************
             Name: 10.17.12.159_9010_1630293077685
               IP: 10.17.12.159
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: false
        ClusterId: 1583270145
             Join: true
            Alive: true
ReplayedJournalId: 285
    LastHeartbeat: 2021-08-30 11:16:50
         IsHelper: true
           ErrMsg: 
          Version: 0.14.13-release-Unknown
3 rows in set (0.03 sec)

mysql> 

6、启动BE集群

（1）添加BE节点

mysql> ALTER SYSTEM ADD BACKEND "node1:9050";
Query OK, 0 rows affected (0.04 sec)

mysql> ALTER SYSTEM ADD BACKEND "node2:9050";
Query OK, 0 rows affected (0.02 sec)

mysql> ALTER SYSTEM ADD BACKEND "node3:9050";
Query OK, 0 rows affected (0.01 sec)

mysql>

（2）查看BE集群状态

mysql> show backends \G
*************************** 1. row ***************************
            BackendId: 10002
              Cluster: default_cluster
                   IP: 10.17.12.158
        HeartbeatPort: 9050
               BePort: -1
             HttpPort: -1
             BrpcPort: -1
        LastStartTime: NULL
        LastHeartbeat: NULL
                Alive: false
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: .000 
        AvailCapacity: 1.000 B
        TotalCapacity: .000 
              UsedPct: 0.00 %
       MaxDiskUsedPct: 0.00 %
               ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)
              Version: 
               Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
*************************** 2. row ***************************
            BackendId: 10003
              Cluster: default_cluster
                   IP: 10.17.12.159
        HeartbeatPort: 9050
               BePort: -1
             HttpPort: -1
             BrpcPort: -1
        LastStartTime: NULL
        LastHeartbeat: NULL
                Alive: false
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: .000 
        AvailCapacity: 1.000 B
        TotalCapacity: .000 
              UsedPct: 0.00 %
       MaxDiskUsedPct: 0.00 %
               ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)
              Version: 
               Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
*************************** 3. row ***************************
            BackendId: 10004
              Cluster: default_cluster
                   IP: 10.17.12.160
        HeartbeatPort: 9050
               BePort: -1
             HttpPort: -1
             BrpcPort: -1
        LastStartTime: NULL
        LastHeartbeat: NULL
                Alive: false
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: .000 
        AvailCapacity: 1.000 B
        TotalCapacity: .000 
              UsedPct: 0.00 %
       MaxDiskUsedPct: 0.00 %
               ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)
              Version: 
               Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
3 rows in set (0.01 sec)

mysql> 

这是因为当前3个BE节点都没有启动

（3）启动BE节点

[root@node1 ~]# cd /app/doris-0.14.13/be
[root@node1 be]# sh bin/start_be.sh --daemon
[root@node1 be]#  ps -aux |grep _be
root     11402  2.3  1.3 2875316 111728 pts/2  Sl   11:23   0:00 /app/doris-0.14.13/be/lib/palo_be
root     12344  0.0  0.0 112664   956 pts/2    S+   11:23   0:00 grep --color=auto _be
[root@node1 be]# 

[root@node2 be]# pwd
/app/doris-0.14.13/be
[root@node2 be]# sh bin/start_be.sh --daemon
[root@node2 be]# ps -aux |grep _be
root     27454  0.6  1.4 2876336 114208 pts/0  Sl   11:21   0:00 /app/doris-0.14.13/be/lib/palo_be
root     29140  0.0  0.0 112664   960 pts/0    S+   11:22   0:00 grep --color=auto _be
[root@node2 be]#

[root@node3 be]# pwd
/app/doris-0.14.13/be
[root@node3 be]# sh bin/start_be.sh --daemon
[root@node3 be]# ps -aux |grep _be
root     15805  1.6  1.1 2876336 95692 pts/0   Sl   11:22   0:00 /app/doris-0.14.13/be/lib/palo_be
root     16473  0.0  0.0 112668   960 pts/0    S+   11:22   0:00 grep --color=auto _be
[root@node3 be]# 

（4）稍等片刻，再次查看BE集群状态

mysql> show backends \G
*************************** 1. row ***************************
            BackendId: 10002
              Cluster: default_cluster
                   IP: 10.17.12.158
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2021-08-30 11:23:50
        LastHeartbeat: 2021-08-30 11:25:56
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: .000 
        AvailCapacity: 6.456 GB
        TotalCapacity: 49.090 GB
              UsedPct: 86.85 %
       MaxDiskUsedPct: 86.85 %
               ErrMsg: 
              Version: 0.14.13-release-Unknown
               Status: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:52","lastStreamLoadTime":-1}
*************************** 2. row ***************************
            BackendId: 10003
              Cluster: default_cluster
                   IP: 10.17.12.159
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2021-08-30 11:21:53
        LastHeartbeat: 2021-08-30 11:25:56
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: .000 
        AvailCapacity: 9.921 GB
        TotalCapacity: 49.090 GB
              UsedPct: 79.79 %
       MaxDiskUsedPct: 79.79 %
               ErrMsg: 
              Version: 0.14.13-release-Unknown
               Status: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:56","lastStreamLoadTime":-1}
*************************** 3. row ***************************
            BackendId: 10004
              Cluster: default_cluster
                   IP: 10.17.12.160
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2021-08-30 11:22:04
        LastHeartbeat: 2021-08-30 11:25:56
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: .000 
        AvailCapacity: 7.602 GB
        TotalCapacity: 49.090 GB
              UsedPct: 84.51 %
       MaxDiskUsedPct: 84.51 %
               ErrMsg: 
              Version: 0.14.13-release-Unknown
               Status: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:07","lastStreamLoadTime":-1}
3 rows in set (0.01 sec)

mysql>