Airflow 集群安装

1.主机规划

内网ip & 主机名 &角色	软件 & 运行进程
192.168.18.124 (master1)	webserver, scheduler
192.168.18.125 (master2)	webserver
192.168.18.126	worker
192.168.18.127	worker
192.168.18.128	worker

2.master1 修改airflow配置 并完成相应的airflow初始化

参考<<Airflow 调度框架 安装经验分享 单节点>>

注意这里的mysql和redis链接地址需要改为相应服务器ip的地址
# 执行步骤1-8, 
# 9,10先不要执行

3.在master1 安装master的高可用组件

pip3 install git+git://github.com/teamclairvoyant/airflow-scheduler-failover-controller.git@v1.0.5
# 初始化
scheduler_failover_controller init

# 修改配置文件 airflow.cfg
# 需要修改
# scheduler_nodes_in_cluster = localhost
scheduler_nodes_in_cluster = 192.168.18.124,192.168.18.125

4.验证

scheduler_failover_controller test_connection

5.将配置ariflow文件发送到相应服务器并启动各组件

cd /usr/local
scp -r airflow 192.168.18.125:`pwd`
scp -r airflow 192.168.18.126:`pwd`
scp -r airflow 192.168.18.127:`pwd`
scp -r airflow 192.168.18.128:`pwd`

# master1
airflow webserver -D
airflow scheduler -D
# master2
airflow webserver -D

# worker1
airflow worker -D
# worker2
airflow worker -D
# worker3
airflow worker -D

6.启动高可用组件

nohup scheduler_failover_controller start > /dev/null &

参考 https://www.lmlphp.com/user/7265/article/item/354892/