azkaban源码编译&centos部署

1. 下载源码

• git clone https://github.com/azkaban/azkaban.git
• 如果下载速度过慢，建议使用gitee加速，gitee加速可参考 https://51.ruyo.net/15373.html

2. 安装配置gradle

• gradle下载地址 https://gradle.org/releases/
• 需要下载项目对应版本的gradle，重用mvn 本地仓库

新增文件 D:\soft\gradle-5.0-all\gradle-5.0\init.d\init.gradle
allprojects {
    repositories {
        maven { url 'file:///D:/soft/apache-maven-3.6.1-bin/apache-maven-3.6.1/repo'}
        mavenLocal()
        maven { name "Alibaba" ; url "https://maven.aliyun.com/repository/public" }
        maven { name "Bstek" ; url "http://nexus.bsdn.org/content/groups/public/" }
        mavenCentral()
    }

    buildscript { 
        repositories {		
            maven { name "Alibaba" ;url 'https://maven.aliyun.com/repository/public' }
            maven { name "Bstek" ;url 'http://nexus.bsdn.org/content/groups/public/' }
            maven { name "M2" ; url 'https://plugins.gradle.org/m2/' }
        }
    }
}

3. 编译源码

• 安装好gradle后进入azkaban源码根目录，并运行gradle命令

cd /source/azkaban
./gradlew clean build installDist -x test

4. 复制以下编译后文件准备部署

1. D:\workspace\idea\clone\gitee\azkaban\azkaban-web-server\build\distributions\azkaban-web-server-3.91.0-283-ga7168bd4.tar.gz
2. D:\workspace\idea\clone\gitee\azkaban\azkaban-exec-server\build\distributions\azkaban-exec-server-3.91.0-283-ga7168bd4.tar.gz
3. D:\workspace\idea\clone\gitee\azkaban\azkaban-db\build\distributions\azkaban-db-3.91.0-283-ga7168bd4.tar.gz
4. D:\workspace\idea\clone\gitee\azkaban\azkaban-solo-server\build\distributions\azkaban-solo-server-3.91.0-283-ga7168bd4.tar.gz

5. Azikaban有三种部署模式

• 下面的例子是使用的第二种模式部署

1. solo-server模式:内嵌一个H2,Webserver和ExecuteServer在一个进程里面.一般用于学习和测试
2. two-server模式,DB使用MySQL,MySQL支持主从架构,Webserver和ExecuteServer运行在不同进程里
3. multiple-executor模式(分布式),DB同上,但Webserver和ExcuteServer运行在不同机器上面,且可能有多个ExcuteServer

6. 上传第4步准备的文件并解压

1. cd /home/soft/
   mkdir azkaban3.91.0
   cd azkaban3.91.0/
2. 上传下列文件到centos新建目录
   azkaban-web-server-3.91.0-283-ga7168bd4.tar.gz
   azkaban-exec-server-3.91.0-283-ga7168bd4.tar.gz
   azkaban-db-3.91.0-283-ga7168bd4.tar.gz
3. 解压文件
   tar -zxvf azkaban-web-server-3.91.0-283-ga7168bd4.tar.gz
   tar -zxvf azkaban-exec-server-3.91.0-283-ga7168bd4.tar.gz
   tar -zxvf azkaban-db-3.91.0-283-ga7168bd4.tar.gz
   mv azkaban-web-server-3.91.0-283-ga7168bd4 server
   mv azkaban-exec-server-3.91.0-283-ga7168bd4 executor

7. 修改server/conf/azkaban.properties

• 例子如下：

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/home/soft/azkaban3.91.0/server/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/home/soft/azkaban3.91.0/server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
executor.port=12321
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=192.168.137.1
mysql.database=azkaban3.91
mysql.user=root
mysql.password=root
mysql.numconnections=100
mysql.serverTimezone=Asia/Shanghai
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

8. 修改executor/conf/azkaban.properties

• 例子如下：

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/home/soft/azkaban3.91.0/server/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/home/soft/azkaban3.91.0/server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/home/soft/azkaban3.91.0/server/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Where the Azkaban web server is located
azkaban.webserver.url=http://localhost:8081
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/home/soft/azkaban3/executor/plugins/jobtypes
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=192.168.137.1
mysql.database=azkaban3.91
mysql.user=root
mysql.password=root
mysql.numconnections=100
mysql.serverTimezone=Asia/Shanghai
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30
azkaban.executor.runtimeProps.override.eager=false
executor.port=12321

9. 初始化mysql数据库

• 进入mysql，创建azkaban数据库
• 运行sql建azkaban系统表，例如：

source /home/soft/azkaban3.91.0/azkaban-db/create-all-sql-3.91.0-283-ga7168bd4.sql

• 注意mysql的时区，此版本azkaban已使用mysql8.x jdbc，增加了时区（serverTimezone）属性，一定要确保mysql中timeZone与azkaban中的配置一致
• mysql 查询设置时区命令参考

SHOW VARIABLES LIKE "%time_zone%";
SET GLOBAL time_zone = '+8:00';
SET time_zone = '+8:00';

10. 去除windows下生成脚本中的换行符

sed -i 's/\r//g' /home/soft/azkaban3.91.1/executor/bin/start-exec.sh
sed -i 's/\r//g' /home/soft/azkaban3.91.1/executor/bin/shutdown-exec.sh
sed -i 's/\r//g' /home/soft/azkaban3.91.1/executor/bin/internal/internal-start-executor.sh
sed -i 's/\r//g' /home/soft/azkaban3.91.1/executor/bin/internal/util.sh

sed -i 's/\r//g' /home/soft/azkaban3.91.1/server/bin/start-web.sh
sed -i 's/\r//g' /home/soft/azkaban3.91.1/server/bin/shutdown-web.sh 
sed -i 's/\r//g' /home/soft/azkaban3.91.1/server/bin/internal/internal-start-web.sh
sed -i 's/\r//g' /home/soft/azkaban3.91.1/server/bin/internal/util.sh

11. 按顺序依次启动executor，激活executor，再启动server

• 例子如下:

1. cd /home/soft/azkaban3.91.0/executor/bin
2. ./start-exec.sh
3. 使用浏览器访问指定url激活执行器，http://192.168.137.10:12321/executor?action=activate
4. cd /home/soft/azkaban3.91.0/server/bin
5. ./start-web.sh

12. 访问web url

• 端口见server/conf/azkaban.properties，例子:http://192.168.137.10:8081/
• 默认密码 azkaban/azkaban