azkaban安装教程

Azkaban是一个任务调度框架
官网下载地址(需要挂梯子)
官方英文文档

0.前置准备条件

1.你的jdk版本至少在1.8（含）以上
2.服务器需要安装mysql
3.服务器安装了git

创建mysql的数据库

mysql> CREATE DATABASE azkaban;
mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;

默认属性max_allowed_pa​​cket太小了，修改/etc/my.cnf

[mysqld]
...
max_allowed_packet=1024M

1.解压azkaban

tar -zxvf azkaban-3.90.0.tar.gz
mv azkaban-3.90.0 /usr/local/

解压完成后目录如下

2.通过gradlew 编译

在azkaban目录下测试build

./gradlew build

编译成功后进入/usr/local/azkaban-3.90.0/azkaban-db/build/distributions解压sql

[root@hadoop001 distributions]# ll
total 28
-rw-r--r-- 1 root root  6447 Sep 15 17:07 azkaban-db-0.1.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 16799 Sep 15 17:07 azkaban-db-0.1.0-SNAPSHOT.zip

解压sql

unzip azkaban-db-0.1.0-SNAPSHOT.zip

导入sql

create database azkaban;
use azkaban;
source /usr/local/azkaban-3.90.0/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql

3.拷贝出编译好的exec-server web-server

cd /usr/local/azkaban-3.90.0
cp -r azkaban-exec-server/build/install/azkaban-exec-server/ /usr/local/
cp -r azkaban-web-server/build/install/azkaban-web-server/ /usr/local/

4.配置并启动exec-server

修改azkaban.properties的配置添加mysql等内容

cd /usr/local/azkaban-exec-server/conf

# 时间改为中国时区
default.timezone.id=Asia/Shanghai

# 这里配置文件一定要修改为绝对路径
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/usr/local/azkaban-exec-server/conf/azkaban-users.xml

# 这里配置文件一定要修改为绝对路径
executor.global.properties=/usr/local/azkaban-exec-server/conf/global.properties

# 修改mysql账户密码
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

进入/bin启动并激活

./start-exec.sh

启动后会生成executor.port的端口号

[root@hadoop001 bin]# ll
total 24
drwxr-sr-x 2 root root    6 Sep 16 09:11 executions
-rw-r--r-- 1 root root    5 Sep 16 09:11 executor.port
-rw-r--r-- 1 root root 8313 Sep 16 09:14 executorServerLog__2020-09-16+09:11:00.out
drwxr-xr-x 2 root root   55 Sep 15 17:58 internal
drwxr-xr-x 2 root root   36 Sep 16 09:11 logs
drwxr-xr-x 2 root root    6 Sep 16 09:11 projects
-rwxr-xr-x 1 root root  214 Sep 15 17:58 shutdown-exec.sh
-rwxr-xr-x 1 root root  207 Sep 15 17:58 start-exec.sh
drwxr-xr-x 2 root root    6 Sep 16 09:11 temp

官方推荐使用curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo激活

[root@hadoop001 bin]# cat executor.port 
42436
[root@hadoop001 bin]# curl -G "localhost:42436/executor?action=activate" && echo
{"status":"success"}

4.配置并启动web-server

进入conf目录修改配置

cd /usr/local/azkaban-web-server/conf

# 时间改为中国时区
default.timezone.id=Asia/Shanghai

# 这里配置文件一定要修改为绝对路径
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/usr/local/azkaban-web-server/conf/azkaban-users.xml
# 这里配置文件一定要修改为绝对路径
executor.global.properties=/usr/local/azkaban-web-server/conf/global.properties

# css样式文件修改绝对路径
web.resource.dir=/usr/local/azkaban-web-server/web/

# 修改mysql配置
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

# 这里机器剩余内存不足6G需要删除MinimumFreeMemory否则报错！！！！
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

到/bin启动web-server

./start-web.sh

Azkaban报错-azkaban.executor.ExecutorManagerException: No active executors found
修改azkaban数据库的executors 表中active=1

查看是否启动成功，启动失败查看log排查

[root@hadoop001 bin]# lsof -i:8081
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    20100 root   90u  IPv6 6678237      0t0  TCP *:tproxy (LISTEN)