使用supervisor实现Apache Doris进程自动拉起

工作变成艺术

已于 2023-08-03 15:56:42 修改

阅读量693

点赞数

分类专栏： Doris 文章标签： apache 数学建模

于 2023-06-21 16:17:12 首次发布

本文链接：https://blog.csdn.net/S1124654/article/details/131328278

版权

Doris 专栏收录该内容

10 篇文章 5 订阅

订阅专栏

该文介绍了如何使用Supervisor自动化管理ApacheDoris的BE、FE和Broker节点。首先，需安装并启动Supervisor服务。接着，修改Doris各节点的启动脚本，移除后台运行符号。然后，为每个节点创建Supervisor配置文件，设置自动启动和重启选项。最后，通过Supervisor的控制命令管理这些进程，确保其稳定运行。

摘要由CSDN通过智能技术生成

参考文章

使用supervisor实现Apache Doris进程自动拉起

操作

自动拉起BE节点

1.使用yum命令安装(推荐)
show backends;
sudo  yum install epel-release
sudo  yum install -y supervisor
sudo  systemctl enable supervisord # 开机自启动
sudo  systemctl start supervisord # 启动supervisord服务
sudo  systemctl status supervisord # 查看supervisord服务状态
sudo  ps -ef|grep supervisord # 查看是否存在supervisord进程

2.配置doris be 进程管理
1.1修改各个 start_be.sh 脚本，去掉最后的 & 符号
vi /home/bigdata/apache-doris-be-1.2.0-bin-x86_64/bin/start_be.sh
ulimit -n 524288
nohup ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" >>"${LOG_DIR}/be.out" 2>&1 </dev/null &
修改成nohup ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" >>"${LOG_DIR}/be.out" 2>&1 </dev/null
wq保存退出
1.2创建be supervisor进程管理配置文件
sudo vi /etc/supervisord.d/doris_be.ini

[program:doris_be]      
process_name=%(program_name)s      
directory=/home/bigdata/apache-doris-be-1.2.0-bin-x86_64
command=sh /home/bigdata/apache-doris-be-1.2.0-bin-x86_64/bin/start_be.sh
autostart=true
autorestart=true
user=root
numprocs=1
startretries=3
stopasgroup=true
killasgroup=true
startsecs=5
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 10
stdout_logfile=/var/log/supervisor-doris_be.log 

3.重新加载配置
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl status
sudo supervisorctl start <process_name> sudo supervisorctl start doris_be
sudo supervisorctl stop <process_name> sudo supervisorctl stop doris_be
sudo supervisorctl restart <process_name> sudo supervisorctl restart doris_be

自动拉起FE

3 配置fe进程管理
show frontends;
##使用yum命令安装(推荐)
sudo  yum install epel-release
sudo  yum install -y supervisor
sudo  systemctl enable supervisord # 开机自启动
sudo  systemctl start supervisord # 启动supervisord服务
sudo  systemctl status supervisord # 查看supervisord服务状态
sudo  ps -ef|grep supervisord # 查看是否存在supervisord进程

##修改各个 start_fe.sh 脚本，去掉最后的 & 符号
vi /home/bigdata/module/apache-doris-fe-1.2.0-bin-x86_64/bin/start_fe.sh
ulimit -n 524288
147行nohup $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" >> $LOG_DIR/fe.out 2>&1 </dev/null &
修改为nohup $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" >> $LOG_DIR/fe.out 2>&1 </dev/null
3.2 创建fe supervisor进程管理配置文件
sudo vi /etc/supervisord.d/doris_fe.ini

[program:dorisfe]
environment = JAVA_HOME="/opt/java/jdk1.8.0_191"
process_name=dorisfe
directory=/home/bigdata/module/apache-doris-fe-1.2.0-bin-x86_64
command=sh /home/bigdata/module/apache-doris-fe-1.2.0-bin-x86_64/bin/start_fe.sh
autostart=true
autorestart=true
user=root
numprocs=1
startretries=3
stopasgroup=true
killasgroup=true
startsecs=10
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 10
stdout_logfile=/var/log/supervisor-doris_fe.log 

3.重新加载配置
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl status
sudo supervisorctl start <process_name> sudo supervisorctl start dorisfe
sudo supervisorctl stop <process_name> sudo supervisorctl stop dorisfe
sudo supervisorctl restart <process_name> sudo supervisorctl restart dorisfe

自动拉起Broke

重启broker;

show broker;
sudo vi /etc/supervisord.conf
minfds=65535

2.配置broker进程管理
2.1 修改各个 start_broker.sh 脚本，去掉最后的 & 符号
vi /home/bigdata/module/apache-doris-dependencies-1.2.4.1-bin-x86_64/apache_hdfs_broker/bin/start_broker.sh
83行修改 nohup $LIMIT $JAVA $JAVA_OPTS org.apache.doris.broker.hdfs.BrokerBootstrap "$@" >> $BROKER_LOG_DIR/apache_hdfs_broker.out 2>&1 </dev/null &
为nohup $LIMIT $JAVA $JAVA_OPTS org.apache.doris.broker.hdfs.BrokerBootstrap "$@" >> $BROKER_LOG_DIR/apache_hdfs_broker.out 2>&1 </dev/null
2.2创建broker supervisor进程管理配置文件
sudo vi /etc/supervisord.d/doris_broker.ini

[program:BrokerBootstrap]
environment = JAVA_HOME="/home/bigdata/module/jdk1.8.0_161"
process_name=%(program_name)s
directory=/home/bigdata/module/apache-doris-dependencies-1.2.4.1-bin-x86_64/apache_hdfs_broker
command=sh /home/bigdata/module/apache-doris-dependencies-1.2.4.1-bin-x86_64/apache_hdfs_broker/bin/start_broker.sh
autostart=true
autorestart=true
user=root
numprocs=1
startretries=3
stopasgroup=true
killasgroup=true
startsecs=5
redirect_stderr=true
stdout_logfile_maxbytes=20MB
stdout_logfile_backups=10
stdout_logfile=/var/log/supervisor-BrokerBootstrap.log

3.重新加载配置
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl status
sudo supervisorctl start <process_name> sudo supervisorctl start BrokerBootstrap
sudo supervisorctl stop <process_name> sudo supervisorctl stop BrokerBootstrap
sudo supervisorctl restart <process_name> sudo supervisorctl restart BrokerBootstrap