azkaban安装(Linux环境)

1. 获取源码: Release 3.90.0 · azkaban/azkaban · GitHub 这里我安装的版本是3.9
2. 解压编译源码,操作如下:

    注意: ❶ mysql版本和mysql-connector-java.jar版本对应关系 ❷ mysql版本和驱动class类名对应关系 否则可能导致启动失败

    可能会有如下报错:

     ①java.sql.SQLException: java.lang.ClassCastException: class java.math.BigInteger cannot be cast to class java.lang.Long

        mysqlConnector       : 'mysql:mysql-connector-java:5.1.28',源码中mysql-connector-java版本是5.1.28,和我的mysql server 8.0.24版本不兼容

     ②ERROR [StdOutErrRedirect] [main] [Azkaban] Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'.

       The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.

        com.mysql.jdbc.Driver是mysql-connector-java 5中的，而com.mysql.cj.jdbc.Driver是mysql-connector-java 6+的

    [root@iZwz9fv6t02o8d1osh1dxgZ ~]# mysql -V
    mysql  Ver 8.0.24 for Linux on x86_64 (Source distribution) 这里的mysql版本是8.0.24

   所以需要修改源码中两处:

   ● 源码 azkaban-3.90.0/build.gradle中: mysqlConnector       : 'mysql:mysql-connector-java:8.0.15' # 版本修改为8.0.15

   ● 源码 azkaban-3.90.0/azkaban-db/src/main/java/azkaban/db/MySQLDataSource.java文件中将setDriverClassName("com.mysql.cj.jdbc.Driver");

※编译源码: gradlew build installDist -x test

生成文件如下:(tar.gz和zip) 

  /www/xhserve/env/azkaban-3.90.0/azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz  #server端项目

  /www/xhserve/env/azkaban-3.90.0/azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz    #web端项目

分别将两个项目解压,tar.gz和zip任选其一

※建库建表用来存储元信息:(这里采用mysql持久存储)

    ❶建库:   mysql> CREATE DATABASE azkaban; (库名随意)

    ❷创建用户赋权限: 
      mysql> CREATE USER '用户名'@'%' IDENTIFIED BY '密码';
      mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON 库名.* to '用户名'@'%' WITH GRANT OPTION;

    ❸建表: 执行脚本.sql,脚本位于编译好的azkaban-db项目中,具体路径参考如下

       mysql> /www/xhserve/env/azkaban-3.90.0/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql
※修改配置
    ❶ server

 # Azkaban Personalization Settings
    azkaban.name=web界面标题,自定义
    azkaban.label=副标题,自定义
    azkaban.color=#FF3601
    azkaban.default.servlet.path=/index #web访问路径
    web.resource.dir=/www/xhserve/env/azkaban-3.9/azkaban-web-server/web/ #web根目录,建议使用绝对地址
    default.timezone.id=Asia/Shanghai #配置时区
    # Azkaban UserManager class
    user.manager.class=azkaban.user.XmlUserManager
    user.manager.xml.file=/www/xhserve/env/azkaban-3.9/azkaban-web-server/conf/azkaban-users.xml #web端用户信息配置文件,建议绝对路径
    # Loader for projects
    executor.global.properties=/www/xhserve/env/azkaban-3.9/azkaban-exec-server/conf/global.properties # sever端,绝对路径
    azkaban.project.dir=projects
    # Velocity dev mode
    velocity.dev.mode=false
    # Azkaban Jetty server properties.
    jetty.use.ssl=false
    jetty.maxThreads=25
    jetty.port=12321 #前端端口
    # Where the Azkaban web server is located
    azkaban.webserver.url=http://localhost:12321 #前端访问地址
    # mail settings 邮件配置可选
    mail.sender=
    mail.host=
    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    # when this parameters set then these parameters are used to generate email links.
    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    # azkaban.webserver.external_hostname=myazkabanhost.com
    # azkaban.webserver.external_ssl_port=443
    # azkaban.webserver.external_port=8081
    job.failure.email=
    job.success.email=
    lockdown.create.projects=false
    cache.directory=cache
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban plugin settings 
    azkaban.jobtype.plugin.dir=/www/xhserve/env/azkaban-3.9/azkaban-exec-server/plugins/jobtypes
    # Azkaban mysql settings by default. Users should configure their own username and password.
    database.type=mysql
    mysql.port=3306
    mysql.host=localhost
    mysql.database=你自己的
    mysql.user=你自己的
    mysql.password=你自己的
    mysql.numconnections=100
    # Azkaban Executor settings
    executor.maxThreads=50
    executor.flow.threads=30
    executor.port=12322 #建议这里手动配置executor端口,如果没配置,server端启动会默认随机设置端口

❷ web端

# Azkaban Personalization Settings
azkaban.name=Xhhb Task Scheduler
azkaban.label=Welcome to Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/www/xhserve/env/azkaban-3.9/azkaban-web-server/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/www/xhserve/env/azkaban-3.9/azkaban-web-server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/www/xhserve/env/azkaban-3.9/azkaban-exec-server/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=12321 #前端端口,和服务端要一致
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=你自己的
mysql.user=你自己的
mysql.password=你自己的
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

  注意注意注意:前后端配置稍有区别,路径建议都使用绝对路径,涉及到的端口一定要能通信,如果使用云主机服务器尤其要注意开放端口,否则项目启动后无法访问登录界面
※项目启动:

   ❶ 先起server端,执行启动命令,在哪个路径下启动服务,日志输出就在当前路径下

       [root@iZwz9fv6t02o8d1osh1dxgZ azkaban-3.9]# ./azkaban-exec-server/bin/start-exec.sh

​    

 注意: 别遗漏了如下操作,激活executor,否则web端启动时会报错找不到executor

 [root@iZwz9fv6t02o8d1osh1dxgZ azkaban-3.9]# curl -G "localhost:端口/executor?action=activate" && echo   #端口就是server端启动成功后输出的executor.port中的端口(如果自己手动配置了,这里的端口就是手动配置的端口)
 {"status":"success"}    //激活成功,其实就是修改数据表中对应的数据

❷启动web,执行启动命令

 [root@iZwz9fv6t02o8d1osh1dxgZ azkaban-3.9]# ./azkaban-web-server/bin/start-web.sh

 ※初体验 : http://ip[或者域名]:端口/index

   ​[root@iZwz9fv6t02o8d1osh1dxgZ azkaban-3.9]# cat azkaban-web-server/conf/azkaban-users.xml 
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

登录名/密码: azkaban