Azkaban3.X 多个executors安装文档

Azkaban3.X 多个executors安装文档

1.编译源码

1.1 克隆源代码

git clone https://github.com/azkaban/azkaban.git

1.2 编译

# 进入azkaban目录
cd azkaban; 

# 使用gradle编译项目
./gradlew build installDist

编译完成后，会在如azkaban-executor-server，azkaban-db等目录下生成build目录。

2.mysql配置

2.1 创建一个azkaban的数据库

mysql> create database azkaban;

2.2 创建一个azkaban的用户并授权

mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';

mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;

mysql> flush privileges;

2.3 导入azkaban的创建表的sql

进入到azkaban/azkaban-bd/build/sql目录，将create-all-sql-3.60.0-15-gee6e229.sql文件导入mysql。

# 将数据导入mysql
mysql> use azkaban;
mysql> source /opt/azkaban/azkaban-db/build/sql/create-all-sql-3.60.0-15-gee6e229.sql

3.配置keystore

执行命令，生成keystore文件：

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

这里所有的密码都设置为azkaban，最终会在当前目录下生成一个keystore的文件，将文件复制到azkaban-web-server目录

cp keystore /opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/

4.配置azkaban-exec-server

进入到/opt/azkaban/azkaban-exec-server/build/install/azkaban-exec-server/conf目录，修改azkaban.properties文件

修改后的文件内容：

# Azkaban Personalization Settings
azkaban.name=DevAzkaban
azkaban.label=Dev Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
# jetty.use.ssl=false
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban

# Where the Azkaban web server is located
# azkaban.webserver.url=http://localhost:8081
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/opt/azkaban/azkaban-exec-server/build/install/azkaban-exec-server/plugins/jobtypes
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=dev-bg-01
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30

# Azkaban Executor settings
executor.port=12321

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

这里修改的有：

1.将default.timezone.id=America/Los_Angeles 修改为：default.timezone.id=Asia/Shanghai

2.相对路径改为绝对路径

3.支持jetty的ssl，不支持http

4.executor.port=12321

5.mysql的信息

5.配置azkaban-web-server

# Azkaban Personalization Settings
azkaban.name=DevAnkaban
azkaban.label=Dev Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
# jetty.use.ssl=false
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban

# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=dev-bg-01
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

# Azkaban Executor settings
executor.port=12321

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1        

与azkaban-exec-server的配置相同。

6.启动azkaban-exec-server和azkaban-web-server

6.1 启动azkaban-exec-server

进入到azkaban-exec-server目录，执行：

# 启动azkaban-exec-server服务
> bin/start-exec.sh

# 停止
> bin/shutdown-exec.sh

激活执行程序：

curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo

6.2 启动azkaban-web-server

进入到azkaban-web-server目录，执行：

# 启动azkaban-web-server服务
> bin/start-web.sh

# 停止
> bin/shutdown-web.sh

启动成功后，通过访问url：https://dev-bg-01:8443，进入azkaban的登陆页面，用户名和密码就是在azkaban-web-server/conf目录下的azkaban-users.xml文件中。默认用户名和密码都是azkaban。

7.Azkaban启动遇到的问题

Azkaban的问题主要是在配置文件中没有配对：

7.1 java.lang.IllegalArgumentException

java.lang.IllegalArgumentException: azkaban.use.multiple.executors must be true. Single executor mode is not supported any more.

这个是在配置文件中没有设置azkaban.use.multiple.executors为true：把下面的配置加上就好。

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1   

7.2 java.io.FileNotFoundException

这个异常需要把配置中的相对路径改为绝对路径

如：

user.manager.xml.file=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml

executor.global.properties=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties

azkaban.jobtype.plugin.dir=/opt/azkaban/azkaban-exec-server/build/install/azkaban-exec-server/plugins/jobtypes

7.3 azkaban.executor.ExecutorManagerException: No active executors found

7.4 jetty没有通过keystore，直接通过http访问

直接通过http访问，是可以操作界面的，但是不能执行任务，通过jetty通过keystore配置ssl，才可以执行。

具体配置：

# jetty.use.ssl=false
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban

参考文档：

Azkaban Getting Started

azkaban3.4分布式多executor编译安装