azkaban的安装与使用

---

一、安装

1、上传、解压、重命名

通过网盘分享的文件：azkaban
链接: 提取码: 1234

azkaban是一个zip压缩文件，所以解压时需要用unzip

unzip azkaban-3.56.0.zip -d /opt/installs/

2、在mysql中生成azkaban的元数据

在mysql中创建数据库azkaban

create database azkaban CHARACTER SET UTF8;

解压azkaban-db-0.1.0-SNAPSHOT.zip，执行下面的create-all-sql-0.1.0-SNAPSHOT.sql文件

source /opt/modules/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql;

3、配置web-server

在web-server文件夹下生成keystore文件

cd /opt/installs/azkaban/web-server
keytool -keystore keystore -alias jetty -genkey -keyalg RSA

修改web-server 下的conf 下的 azkaban.properties文件：
将原来的删除，添加如下配置：

# Azkaban Personalization Settings
azkaban.name=MyAzkaban
azkaban.label=MyAzkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/opt/installs/azkaban/web-server/web
# 时区修改为上海时区
default.timezone.id=Asia/Shanghai

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/installs/azkaban/web-server/conf/azkaban-users.xml

# Loader for projects
executor.global.properties=/opt/installs/azkaban/web-server/conf/global.properties
azkaban.project.dir=projects
# 配置mysql数据库连接的
database.type=mysql
mysql.port=3306
mysql.host=node01
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100

# Velocity dev mode
velocity.dev.mode=false

# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081

# 此处是我们的生成的秘钥密码
jetty.keystore=keystore
jetty.password=123456
jetty.keypassword=123456
jetty.truststore=keystore
jetty.trustpassword=123456

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

# mail settings，azkaban的邮箱通知预警
mail.sender=17724839808@163.com
mail.host=smtp.163.com
mail.user=17724839808@163.com
mail.password=ZICOQUKXMWHVIALF
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

# enduser -> myazkabanhost:443 -> proxy -> localhost:8081

# when this parameters set then these parameters are used to generate email links.

# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache

# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
azkaban.native.lib=false
# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

修改web-server/conf 目录下 azkaban-users.xml：
在user和role中间添加如下配置：

<user password="admin" roles="admin,metrics" username="admin" />

4、配置exec-server

修改exec-webserver 下 的conf 下的azkaban.properties：
将原来的删除，然后添加如下配置：

# Azkaban Personalization Settings
azkaban.name=Azkaban
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/opt/installs/azkaban/web-server/web/
default.timezone.id=Asia/Shanghai

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/installs/azkaban/web-server/conf/azkaban-users.xml

# Loader for projects
executor.global.properties=/opt/installs/azkaban/exec-server/conf/global.properties
azkaban.project.dir=projects/
azkaban.execution.dir=execution/
executor.flow.threads=30
flow.num.job.threads=10
job.log.chunk.size=100
job.log.backup.index=10
job.max.Xms=1
job.max.Xmx=2
azkaban.server.flow.max.running.minutes=-1



# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=node01
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30


# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

# JMX stats
jetty.connector.stats=true
executor.connector.stats=true

azkaban.native.lib=lib/
#azkaban.jobtype.plugin.dir=plugins/jobtypes


# uncomment to enable inmemory stats for azkaban
#executor.metric.reports=true
#executor.metric.milisecinterval.default=60000

azkaban.jobtype.plugin.dir=/opt/installs/azkaban/exec-server/plugins/jobtypes

修改exec-webserver 下插件下的一个配置文件：
plugins/jobtypes/commonprivate.properties

set execute-as-user
execute.as.user=false
memCheck.enabled=false

修改 web-server和exec-server lib下的mysql驱动包：

cd /opt/installs/azkaban/exec-server/lib
rm -rf mysql-connector-java-5.1.28.jar
cp /opt/installs/sqoop/lib/mysql-connector-java-8.0.26.jar ./


cd /opt/installs/azkaban/web-server/lib
rm -rf mysql-connector-java-5.1.28.jar
cp /opt/installs/sqoop/lib/mysql-connector-java-8.0.26.jar ./

5、修改所有的.sh 的执行权限

修改web-server下的执行权限：

cd /opt/installs/azkaban/web-server/bin
chmod 777 start-web.sh shutdown-web.sh
cd internal/
chmod 777 internal-start-web.sh util.sh 

修改exec-server下的执行权限：

cd /opt/installs/azkaban/exec-server/bin
chmod 777 start-exec.sh shutdown-exec.sh
cd internal/
chmod 777 internal-start-executor.sh util.sh 

6、编写azkaban启动停止脚本

#!/bin/bash
# 参数个数一个或者两个 not equal to --> ne
if  [ $# -lt 1 ] && [ $# -gt 2 ] 
then
echo -e  "\033[32mUsage1:\033[0m azkaban.sh 1 \033[32mor\033[0m azkaban.sh 0"
echo -e  "\033[32mUsage2:\033[0m azkaban.sh 1 w \033[32mor\033[0m azkaban.sh 0 w"
exit;
fi


# 输入 1 为启动, 输入 0 为关闭
v1="web"
v2="exec"

if [ $1 -eq 1 ]
then
stat="start-"
elif [ $1 -eq 0 ]
then
stat="shutdown-"
else
echo -e "\033[32mUsage:\033[0m azkaban.sh 1 \033[32mor\033[0m azkaban.sh 0"
exit;
fi

cd /opt/installs/azkaban/logs
# 传入第二个参数 w 代表 web-server , e 代表 exec-server 不传第二个默认全部启动/停止
if [ "$2" = 'w' ]
then
path1=/opt/installs/azkaban/${v1}-server/bin/${stat}${v1}.sh
echo -e "\033[32m${stat}${v1}.sh\033[0m"
sh  ${path1}
elif [ "$2" = 'e' ]
then
path2=/opt/installs/azkaban/${v2}-server/bin/${stat}${v2}.sh
echo -e "\033[32m${stat}${v2}.sh\033[0m"
sh  ${path2}
elif [ 'a'"$2"'a' = 'aa' ]
then
path1=/opt/installs/azkaban/${v1}-server/bin/${stat}${v1}.sh
echo -e "\033[32m${stat}${v1}.sh\033[0m"
sh  ${path1}
sleep 2
path2=/opt/installs/azkaban/${v2}-server/bin/${stat}${v2}.sh
echo -e "\033[32m${stat}${v2}.sh\033[0m"
sh  ${path2}
else
echo -e "\033[32mUsage1:\033[0m azkaban.sh 1 w \033[32mor\033[0m azkaban.sh 0 w"
echo -e "\033[32mUsage2:\033[0m azkaban.sh 1 e \033[32mor\033[0m azkaban.sh 0 e"
exit;
fi

将脚本放到/usr/local/sbin下，并赋权限：

chmod 777 azkaban.sh

创建存放启动和停止azkaban的日志：

cd /opt/installs/azkaban/
# 用于存储两个服务的日志
mkdir logs 

使用：

# 1.启动azkaban
azkaban.sh 1
# 单独启动Web
azkaban.sh 1 w
# 单独启动exec
azkaban.sh 1 e

# 2.关闭azkaban
 azkaban.sh 0
# 单独关闭Web
azkaban.sh 0 w
# 单独关闭exec
azkaban.sh 0 e

二、用法

需要创建两个文件：一个以 .project为后缀，一个以.flow为后缀
在.project文件中编写：

azkaban-flow-version: 2.0

在flow中，编写任务：

nodes:
  - name: jobC
    type: command
    config:
      command: echo "This is job C"
    dependsOn:
      - test_flow
  - name: test_flow
    #此处是type=flow，内嵌流
    type: flow
    config:
      prop: ${canshu}
    #此处有nodes
    nodes:
      - name: jobB
        type: command
        config:
          command: echo "This is job B ${prop}"
        dependsOn:
          - jobA
      - name: jobA
        type: command
        config:
          command: echo "This is job A"