工具：安装定时任务流管理工具Azkaban

第⼀章 mysql配置

1.1. 建库建表
 

# 登录 
mysql -u cdh -p -h 172.16.195.221 cdh rootcdh@2018 

#建库 
mysql> create database azkaban; 

#登库 
mysql> use azkaban; 

#建表 
mysql> source /opt/app/azkaban-db/build/sql/create-all-sql-0.1.0-SNAPSHOT.sql;

1.2. 新增用户

#新建用户
mysql> grant all privileges on azkaban.* to 'azkaban'@'%' identified by
'azkaban';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH
GRANT OPTION;
mysql> insert into executors(host,port)values("pf-bigdata-cdh",12321);
mysql> update executors set active=1;
#刷新权限
mysql> flush privileges;
# 登录
mysql -uroot -p -h 172.16.195.221
mysql -uroot -p -S /huayong/data/mysql/3306/mysql.sock
rootcdh@2018
mysql -uazkaban -p -S /huayong/data/mysql/3306/mysql.sock
azkaban

第二章

安装azkaban-web

2.1 成SSL证书 密码均设为'azkaban'，其他信息随意填写，生成的keystore⽂件复制到conf和bin目录下

[root@pf-bigdata1 azkaban-web-server-0.1.0-SNAPSHOT]# keytool -keystore

keystore -alias jetty -genkey -keyalg RSA

cp keystore bin/

cp keystore conf/

2.2 修改 azkaban.properties

将所有路径都配置为绝对路径，否则会有各种找不到文件的错误

# Azkaban Personalization Settings
azkaban.name=HuaYong
azkaban.label=RUYU Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/opt/app/azkaban-web-server-0.1.0-SNAPSHOT/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/app/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkabanusers.xml
# Loader for projects
executor.global.properties=/opt/app/azkaban-web-server-0.1.0-
SNAPSHOT/conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=172.16.195.221
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
#database.type=mysql
#h2.path=./h2
#h2.create.tables=true
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
jetty.keystore=/opt/app/keystore

jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=/opt/app/keystore
jetty.trustpassword=azkaban
# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=lupeng@hua-yong.com
mail.host=smtp.exmail.qq.com
mail.user=lupeng@hua-yong.com
mail.password=Lu2018
# User facing web server configurations used to construct the user facing
server URLs. They are useful when there is a reverse proxy between Azkaban
web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email
links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl
configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=18505565928m0@sina.cn
job.success.email=18505565928m0@sina.cn
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes

#Multiple Executor Mode
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,Cpu
Status
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

2.3 启动 azkaban-web 启动后在bin⽬录下会产生日志文件，若启动错误可参看工作台输出和日志文件

#启动
nohup /opt/app/azkaban-web-server-0.1.0-SNAPSHOT/bin/internal/internal-startweb.sh
1 > /opt/app/azkaban-web-server-0.1.0-SNAPSHOT/azwebstd.out
2>/opt/app/azkaban-web-server-0.1.0-SNAPSHOT/azweberr.out &
#若需关闭
/opt/app/azkaban-web-server-0.1.0-SNAPSHOT/bin/shutdown-web.sh

2.4 登陆web页面

浏览器输入：http://********:8081/ 用户名：azkaban 密码：azkaban

 

第三章 安装azkaban-exec

3.1 配置azkaban.properties

#Azkaban
default.timezone.id=Asia/Shanghai
# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=/opt/app/azkaban-exec-server-0.1.0-
SNAPSHOT/plugins/jobtypes
#Loader for projects
executor.global.properties=/opt/app/azkaban-exec-server-0.1.0-
SNAPSHOT/conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=172.16.195.221
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

3.2 启动azkaban-exec

启动后在bin⽬录下会产生日志文件，若启动错误可参看工作台输出和日志文件

#启动
nohup /opt/app/azkaban-exec-server-0.1.0-SNAPSHOT/bin/internal/internal-startexecutor.sh
1 > /opt/app/azkaban-exec-server-0.1.0-SNAPSHOT/azwebstd.out
2>/opt/app/azkaban-exec-server-0.1.0-SNAPSHOT/azweberr.out &
#若需关闭
/opt/app/azkaban-exec-server-0.1.0-SNAPSHOT/bin/shutdown-exec.sh

第四章

常见错误

1.azkaban-users.xml找不到 Caused by: java.lang.IllegalArgumentException: User xml file conf/azkaban-users.xml doesn't exist. 解决：配置⽂件azkaban.properties⾥user.manager.xml.file改为绝对路径

2.Keystore找不到 java.io.FileNotFoundException: /opt/azkaban-web-2.5.0/bin/keystore (No such file or directory) 解决：复制keystore⽂件到bin目录下

3.插件地址不存在 ERROR [AzkabanWebServer] [Azkaban] plugin path plugins/triggers doesn't exist! 解决：暂不需要更改

4.网页登陆显示不正常 解决：修改web.resource.dir为绝对路径web.resource.dir=/opt/azkaban-web-2.5.0/web/

5.Azkaban 执行Flow 一直处于 Preparing 状态? 没有错误⽇志输出

6.为什么不在 Executor 启动时自动激活

We don't update prior to web-server start.
Let me introduce how we do a regular Executor deployment. Both old webserver
and old executor(3.a) are up initially, and old executor is running jobs (we
can not interrupt it). Executor Deployment starts:
install new Executor (3.b) on our box without activating.
Run job tests, which targets 3.b and make sure 3.b is working.
Activate 3.b, and deactivate 3.a. Even though 3.a is deactivated, jobs still
keep runnig there.
Deployment completes.
We would better run tests against new executor before activating it. Is it a
bit clear now?

azkaban.executor.ExecutorManagerException: executor became inactive before setting up the flow 2018

curl http://${executorHost}:${executorPort}/executor?action=activate
curl http://****1:12321/executor?action=activate
curl http://****2:12321/executor?action=activate
curl http://****3:12321/executor?action=activate

7.邮件功能突然不能使用 本以为是是25端口的问题，将代码中port改为465，还是不行，然后将host从sina改成qq，ok 原因：sina邮箱由于某种原因不能使用

第五章 使用
 

$ cd /opt/app/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes 

$ vim commonprivate.properties

# set execute-as-user 

execute.as.user=false 

azkaban.native.lib=/opt/app/azkaban-exec-server-0.1.0- SNAPSHOT/plugins/jobtypes 

memCheck.enabled=false 

azkaban.native.lib=false