Azkaban学习日记（三）——multiple-executor模式部署

1、安装所需软件

Azkaban Web 服务安装包

azkaban-web-server-0.1.0-SNAPSHOT.tar.gz

 

Azkaban 执⾏服务安装包

azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz

 

sql 脚本

2、节点规划

linux121 mysql,exec-server

linux122 web-server

linux123 exec-server

3、数据库准备

mysql -u root -p

SET GLOBAL validate_password_length=5;

 

SET GLOBAL validate_password_policy=0;

 

CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';

 

GRANT all privileges ON azkaban.* to 'azkaban'@'%' identified by 'azkaban'

WITH GRANT OPTION;

 

CREATE DATABASE azkaban;

 

use azkaban;

4、linux121解压数据库脚本

[root@linux121 azkaban]# pwd
/opt/lagou/servers/azkaban
[root@linux121 azkaban]# tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz
[root@linux121 azkaban]# cd azkaban-db-0.1.0-SNAPSHOT
[root@linux121 azkaban-db-0.1.0-SNAPSHOT]# ll
total 104
-rw-rw-r-- 1 root root   106 Aug  9  2018 create.active_executing_flows.sql
-rw-rw-r-- 1 root root   265 Aug  9  2018 create.active_sla.sql
-rw-r--r-- 1 root root 11990 May 29  2019 create-all-sql-0.1.0-SNAPSHOT.sql

在linux121数据库

mysql> source /opt/lagou/servers/azkaban/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql; 

5、配置Azkaban-web-server 

分发

[root@linux121 azkaban]# scp azkaban-web-server-0.1.0-SNAPSHOT.tar.gz linux122:/opt/lagou/servers/azkaban
azkaban-web-server-0.1.0-SNAPSHOT.tar.gz 

解压

[root@linux122 azkaban]# tar -zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz

⽣成ssl证书

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

[root@linux122 azkaban-web-server-0.1.0-SNAPSHOT]# ll
total 8
drwxr-xr-x 3 root root   65 May 29  2019 bin
drwxr-xr-x 2 root root  106 May 29  2019 conf
-rw-r--r-- 1 root root 2241 Jun 17 06:55 keystore
drwxr-xr-x 2 root root 4096 May 29  2019 lib
drwxr-xr-x 6 root root   73 May 29  2019 web

 6、修改 azkaban-web-server的配置⽂件

[root@linux122 conf]# pwd
/opt/lagou/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/conf
[root@linux122 conf]# vim azkaban.properties

azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai

jetty.use.ssl=true

# Azkaban mysql settings by default. Users should configure their own username and password.
# KeyStore for SSL ssl相关配置 注意密码和证书路径
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban

database.type=mysql
mysql.port=3306
mysql.host=linux121
mysql.database=azkaban
mysql.user=root
mysql.password=12345678
mysql.numconnections=100

关闭校验#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

添加属性 

mkdir -p plugins/jobtypes

cd plugins/jobtypes/

vim commonprivate.properties

azkaban.native .lib = false

execute.as .user = false

memCheck .enabled = false

7、配置Azkaban-exec-server 

tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz

[root@linux121 conf]# pwd
/opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/conf
[root@linux121 conf]# vim azkaban.properties

default.timezone.id=Asia/Shanghai
azkaban.webserver.url=http://linux122:8443

database.type=mysql
mysql.port=3306
mysql.host=linux121
mysql.database=azkaban
mysql.user=root
mysql.password=12345678
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30
executor.port=12321

8、分发。121的exec-server发到123

[root@linux121 azkaban]# scp -r azkaban-exec-server-0.1.0-SNAPSHOT linux123:/opt/lagou/servers/azkaban 

9、启动

[root@linux121 azkaban-exec-server-0.1.0-SNAPSHOT]# bin/start-exec.sh

[root@linux123 azkaban-exec-server-0.1.0-SNAPSHOT]# bin/start-exec.sh

[root@linux122 azkaban-web-server-0.1.0-SNAPSHOT]# bin/start-web.sh

激活我们的 exec-server

启动 webServer 之后进程失败消失，可通过安装包根⽬录下对应启动⽇志进⾏排查

需要⼿动激活executor

cd /opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT

curl -G "linux121:$(<./executor.port)/executor?action=activate" && echo

curl -G "linux123:$(<./executor.port)/executor?action=activate" && echo 

https://linux122:8443