azkaban3.90.0安装部署

azkaban3.90.0安装部署

写在前面

我采用第三种模式,即一台机器安装webServer服务,多台机器安装execServer.这种模式是使用最广泛的。
1 solo-server模式 (使用内置h2存储元数据)；
2 two-server模式 (1个webServer,1个execServer在同一服务器上,使用mysql存储元数据)；
3 multiple-executor模式 (1个webServer,多个execServer分布在不同服务上,使用mysql存储元数据)；

基础环境

MySQL 配置 [mysql配置很关键]
默认集群已有一个mysql服务器,创建azkaban库,azkaban用户密码,并赋予远程连接。
CREATE DATABASE azkaban;
CREATE USER ‘azkaban’@’%’ IDENTIFIED BY ‘Azkaban-123’;
CREATE USER ‘azkaban’@‘localhost’ IDENTIFIED BY ‘Azkaban-123’;
grant all privileges on azkaban.* to ‘azkaban’@’%’ identified by ‘Azkaban-123’;
grant all privileges on azkaban.* to ‘azkaban’@‘localhost’ identified by ‘Azkaban-123’;
flush privileges;

安装包整理
下载最新azkaban源文件https://github.com/azkaban/azkaban/releases
上传到你的某一个服务器 /opt/ 目录下
1）解压缩文件azkaban-3.90.0.tar.gz
2）进入azkaban-3.90.0目录执行下面代码编译源文件 （编译报错缺少Git命令， yum install -y git）
./gradlew distTar
3）将下面4个文件解压缩,文件如下:
/opt/azkaban-3.90.0/azkaban-web-server/build/distributions/azkaban-web-server-3.90.0-SNAPSHOT.tar.gz
/opt/azkaban-3.90.0/azkaban-exec-server/build/distributions/azkaban-exec-server-3.90.0-SNAPSHOT.tar.gz
/opt/azkaban-3.90.0/azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
/opt/azkaban-3.90.0/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz
4）导入建表语句（azkaban的元数据表）
mysql -uroot -p123456
use azkaban;

SOURCE /usr/local/src/azkaban/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql；

配置exec-server

1、 解压
从/opt/路径里把以下tar包复制到 /usr/local/src/azkaban/
[root@azkaban-3.90.0]# mkdir /usr/local/src/azkaban
[root@azkaban-3.90.0]# cp azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz /usr/local/src/azkaban/packages
[root@azkaban-3.90.0]# cp azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz /usr/local/src/azkaban/packages
[root@azkaban-3.90.0]# cp azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz /usr/local/src/azkaban/packages
[root@azkaban-3.90.0]# cp azkaban-hadoop-security-plugin/build/distributions/azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.tar.gz /usr/local/src/azkaban/packages
[root@azkaban-3.90.0]# cp azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz /usr/local/src/azkaban/packages

解压tar包
[root@azkaban-3.90.0]# cd /usr/local/src/azkaban/
[root@saltstack azkaban]# tar zxf packages/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C .
[root@saltstack azkaban]# tar zxf packages/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C .
[root@saltstack azkaban]# tar zxf packages/azkaban-db-0.1.0-SNAPSHOT.tar.gz -C .
[root@saltstack azkaban]# tar zxf packages/azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.tar.gz -C .
[root@saltstack azkaban]# tar zxf packages/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz -C .
[root@saltstack azkaban]# ls
azkaban-db-0.1.0-SNAPSHOT
azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT
azkaban-web-server-0.1.0-SNAPSHOT
azkaban-exec-server-0.1.0-SNAPSHOT
azkaban-solo-server-0.1.0-SNAPSHOT
packages

2、 配置azkaban.properties
1） 配置key
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
2） 配置web 服务器的properties
[root@node118 ~]# cd /opt/azkaban-3.90.0/
[root@node118 azkaban-3.90.0]# cp keystore azkaban-web-server/

注意：只改黑体加粗部分即可！！！

[root@node118 azkaban]# vim azkaban-exec-server-0.1.0-SNAPSHOT/conf/azkaban.properties

1 # Azkaban Personalization Settings
2 azkaban.name=EDU-AZKABAN
3 azkaban.label=My Local Azkaban
4 azkaban.color=#FF3601
5 azkaban.default.servlet.path=/index
6 web.resource.dir=web/
7 default.timezone.id=Asia/Shanghai
8 # Azkaban UserManager class
9 user.manager.class=azkaban.user.XmlUserManager
10 user.manager.xml.file=conf/azkaban-users.xml
11 # Loader for projects
12 executor.global.properties=conf/global.properties
13 azkaban.project.dir=projects
14 # Velocity dev mode
15 velocity.dev.mode=false
16 # Azkaban Jetty server properties.
17 jetty.use.ssl=false
18 jetty.maxThreads=25
19 jetty.port=8081
20 # Where the Azkaban web server is located
21 azkaban.webserver.url=http://localhost:8081
22 # mail settings
23 mail.sender=
24 mail.host=
25 # User facing web server configurations used to construct the user facing server URLs. They are useful when ther
e is a reverse proxy between Azkaban web servers and users.
26 # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
27 # when this parameters set then these parameters are used to generate email links.
28 # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used
.
29 # azkaban.webserver.external_hostname=myazkabanhost.com
30 # azkaban.webserver.external_ssl_port=443
31 # azkaban.webserver.external_port=8081
32 job.failure.email=
33 job.success.email=
34 lockdown.create.projects=false
35 cache.directory=cache
36 # JMX stats
37 jetty.connector.stats=true
38 executor.connector.stats=true
39 # Azkaban plugin settings
40 #azkaban.jobtype.plugin.dir=plugins/jobtypes
41 azkaban.jobtype.plugin.dir=/usr/local/src/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes
42 # Azkaban mysql settings by default. Users should configure their own username and password.
43 database.type=mysql
44 mysql.port=3306
45 mysql.host=10.126.8.118
46 mysql.database=azkaban
47 mysql.user=azkaban
48 mysql.password=azkaban-123
49 mysql.numconnections=100
50 # Azkaban Executor settings
51 executor.maxThreads=50
52 executor.flow.threads=30

3、 配置用户
vim azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
第四行增加 user username=“admin” password=“admin” roles=“admin,metrics”/>

vim azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes/commonprivate.properties
–set execute-as-user
execute.as.user=false
–新加配置项
azkaban.native.lib=false

配置web-Server

修改conf/azkaban.properties
[root@node118 azkaban]# vim azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban.properties

1 # Azkaban Personalization Settings
2 azkaban.name=EDU-AZKABAN
3 azkaban.label=My Local Azkaban
4 azkaban.color=#FF3601
5 azkaban.default.servlet.path=/index
6 #web.resource.dir=web/
7 web.resource.dir=/usr/local/src/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/web/
8 default.timezone.id=Asia/Shanghai
9 # Azkaban UserManager class
10 user.manager.class=azkaban.user.XmlUserManager
11 #user.manager.xml.file=conf/azkaban-users.xml
12 user.manager.xml.file=/usr/local/src/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
13 # Loader for projects
14 executor.global.properties=conf/global.properties
15 azkaban.project.dir=projects
16 # Velocity dev mode
17 velocity.dev.mode=false
18 # Azkaban Jetty server properties.
19 jetty.use.ssl=false
20 jetty.maxThreads=25
21 jetty.port=8081
22 # Azkaban Executor settings
23 # mail settings
24 mail.sender=
25 mail.host=
26 # User facing web server configurations used to construct the user facing server URLs. They are useful when ther
e is a reverse proxy between Azkaban web servers and users.
27 # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
28 # when this parameters set then these parameters are used to generate email links.
29 # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used
30 # azkaban.webserver.external_hostname=myazkabanhost.com
31 # azkaban.webserver.external_ssl_port=443
32 # azkaban.webserver.external_port=8081
33 job.failure.email=
34 job.success.email=
35 lockdown.create.projects=false
36 cache.directory=cache
37 # JMX stats
38 jetty.connector.stats=true
39 executor.connector.stats=true
40 # Azkaban mysql settings by default. Users should configure their own username and password.
41 database.type=mysql
42 mysql.port=3306
43 mysql.host=10.126.8.118
44 mysql.database=azkaban
45 mysql.user=azkaban
46 mysql.password=azkaban-123
47 mysql.numconnections=100
48 #Multiple Executor
49 azkaban.use.multiple.executors=true
50 azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
51 #azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
52 azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
53 azkaban.executorselector.comparator.Memory=1
54 azkaban.executorselector.comparator.LastDispatched=1
55 azkaban.executorselector.comparator.CpuUsage=1
~
重要说明
以上的操作均是在单节点执行的。但是我们需要1个web和2个exec的集群模式，需要执行以下操作即可：
scp -r /usr/local/src/azkaban 10.126.8.117:/usr/local/src/azkaban
scp -r /usr/local/src/azkaban 10.126.8.119:/usr/local/src/azkaban
然后ABC三台服务器都有一模一样的程序包。至于你想选择哪个当作web，哪2个当作exec,就取决于你去哪启动命令了，启动如下：

启动Azkaban

1、 启动顺序
azkaban集群模式必须先启动executor-server,再启动webServer,不然一定无法运行
2、 启动exec-server
./bin/start-exec.sh
#注意azkaban在启动时默认不会激活executor
#需要手动激活
#mysql所在节点的家目录执行以下命令激活executor
curl -G “localhost:$(<./executor.port)/executor?action=activate” && echo
{“status”:“success”}
#或者进入到Mysql数据库中 更改executors表中的active字段为1 进行激活
然后每个执行节点执行以下命令（很重要、否则会导致job直接failed）
curl http://localhost:port/executor?action=activate
3、 启动web-server
cd azkaban-web-server/
./bin/start-web.sh
登陆 web服务器地址:8081进行登陆,至此azkaban的搭建完成。

邮箱预警

一 vim web-server/conf/azkaban.properties
23 # mail settings
24 mail.sender=@qq.com
25 mail.host=smtp.qq.com
26 mail.user=@qq.com
27 mail.password=授权码（不是qq密码！！！）
二 授权码获取

三 重启 web-server . 亲测成功 ！！【本环境是在阿里云远程配置的azkaban3.90.0版本哦】

欢迎评论！