Azkaban3.80.1 Two Server Mode

Azkaban另一种运行模式the heavier weight two server mode:
数据信息存储在MySQL,用于生产上做主备
webserver和execserver运行在不同的进程

本文是在 <<Azkaban3.80.1 源码编译 solo server环境部署>>的基础上进行安装的

1.Azkaban初始化操作

(1).克隆仓库切换分支
(2).下载相应的依赖包并修改相应的配置
(3).编译安装
(4).安装mysql

2.数据库准备工作

mysql> CREATE DATABASE azkaban;

mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';

mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;

mysql> flush privileges;

# 创建 Azkaban 表
cd /usr/local/azkaban/azkaban-db/build/distributions
tar -zxvf azkaban-db-3.80.1.tar.gz

cd azkaban-db-3.80.1

mysql> use azkaban;

mysql> source /usr/local/azkaban/azkaban-db/build/distributions/azkaban-db-3.80.1/create-all-sql-3.80.1.sql

3.AzkabanExecServer搭建

cd /usr/local/azkaban/azkaban-exec-server/build/distributions
tar -zxvf azkaban-exec-server-3.80.1.tar.gz
cd azkaban-exec-server-3.80.1/conf
# 修改配置(默认就有)
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

4.AzkabanWebServer搭建

cd /usr/local/azkaban/azkaban-web-server/build/distributions
tar -zxvf azkaban-web-server-3.80.1.tar.gz
cd azkaban-web-server-3.80.1/conf

SSL配置
# 生成证书
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
keytool -importkeystore -srckeystore keystore -destkeystore keystore -deststoretype pkcs12

# 修改配置文件
vim azkaban.properties
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

# 解决界面元素无法显示问题 conf/azkaban-users.xml doesn't exist
web.resource.dir=/usr/local/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.80.1/web

# 修改为绝对路径
user.manager.xml.file=/usr/local/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.80.1/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/usr/local/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.80.1/conf/global.properties

# 解决Azkaban3.7执行job时一直处于preparing状态问题,过滤器会检查 executor 主机空余内存是否会大于 6G，如果不足 6G，则 web-server 不会将任务交由该主机执行
#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus

jetty.use.ssl=true
jetty.ssl.port=8443
jetty.maxThreads=25
jetty.port=8081
jetty.keystore=/usr/local/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.80.1/conf/keystore
jetty.password=Cocktail_py
jetty.keypassword=Cocktail_py
jetty.truststore=/usr/local/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.80.1/conf/keystore
jetty.trustpassword=Cocktail_py

5.插件配置Plugin Configurations

cd /usr/local/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.80.1/plugins/jobtypes

# 将execute-as-user.c拷贝到相应的目录
cp /usr/local/azkaban/az-exec-util/src/main/c/execute-as-user.c /usr/local/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.80.1/plugins/jobtypes
gcc execute-as-user.c -o execute-as-user
sudo chown root execute-as-user
sudo chmod 6050 execute-as-user

# 解决Missing required property 'azkaban.native.lib' 问题(版本不同,解决方法有点偏差)
cd /usr/local/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.80.1/conf

vim azkaban.properties
azkaban.jobtype.plugin.dir=/usr/local/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.80.1/plugins/jobtypes

6.启动服务

# 启动exec-server
cd /usr/local/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.80.1
./bin/start-exec.sh
# 激活一个azkaban-exec-server
"""
-bash: ./executor.port: No such file or directory
curl: (7) Failed to connect to ::1: No route to host
# 找到文件所在的位置
find / -name executor.port
"""
cd /usr/local/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.80.1/bin/
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo

# 启动web-server
cd /usr/local/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.80.1
./bin/start-web.sh

# 验证一波
jps

28657 Jps
26997 GradleDaemon
27574 AzkabanWebServer
27212 AzkabanExecutorServer

7.浏览器访问

https://ip:8443

参考: https://azkaban.readthedocs.io/en/latest/getStarted.html#database-setup