centos6.9 安装部署azkaban任务调度工具

1.环境、文件准备

提前安装好适配版本的hadoop、hive（mysql，对应版本jdbc-connector.jar包）

服务器版本：

a.阿里云ECS Centos6.9（master主机）

b.阿里云ECS Ubuntu20.0.4 （hadoopsliver）

hadoop版本：3.2.1

hive版本：3.1.2

mysql-jdbc版本：mysql-connector-java-8.0.28.jar

mysql版本：8.0.29

NOTE：

1.  rpm -qa|grep -i mysql  查看mysql rpm安装包  

2. yum -y remove mysql* 卸载mysql安装包 

3. 删除  find / -name mysql 找到的文件目录

4. rm -rf /etc/my.cnf 删除mysql配置文件，rm -rf /root/.mysql_history rm -rf /root/mysql_secret 删除历史数据和密码

5. yum 安装mysql  

yum search mysql-community

6. 启动mysql服务 

systemctl start mysqld

7. yum安装mysql后，grep 'password' /var/log/mysqld.log查看日志文件默认初始化密码

8. mysql -uroot -p 登录mysql shell

9. alter user 'root'@'localhost' identified by 'Abc12345'; 重置密码；

set global validate_password.policy=0; 修改密码策略为简单

set global validate_password.length=1; 修改密码长度最小值为1

ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456';修改为简单密码和所有IP访问

10. vi /etc/my.cnf 添加mysql配置项

bind-address            = 0.0.0.0  #原127.0.0.1 修改为0.0.0.0
mysqlx-bind-address     = 127.0.0.1

11. mysql -uroot -p123456 登录mysql shell ，OK

下载安装包

azkaban-executor-server-2.5.0.tar.gz

azkaban-sql-script-2.5.0.tar.gz

azkaban-web-server-2.5.0.tar.gz

百度有安装包，不做赘述

2.文件解压

三个安装包都解压到 /usr/local/azkaban 目录下 分别重命名为executor，azkaban，server

3.mysql元数据库配置

进入mysql shell

mysql -uroot -p123456

mysql> create database azkaban;

mysql> use azkaban;

mysql> source /usr/local/azkaban/azkaban/create-all-sql-2.5.0.sql 
# 执行SQL文件，元数据库建表。所有建表语句集中在此SQL文件中。

mysql> show tables;
+------------------------+
| Tables_in_azkaban      |
+------------------------+
| active_executing_flows |
| active_sla             |
| execution_flows        |
| execution_jobs         |
| execution_logs         |
| project_events         |
| project_files          |
| project_flows          |
| project_permissions    |
| project_properties     |
| project_versions       |
| projects               |
| properties             |
| schedules              |
| triggers               |
+------------------------+
15 rows in set (0.00 sec)

即可~

4.修改配置文件

4.1 修改server配置

cd /usr/local/azkaban/server

4.1.1 初始化秘钥

运行

 keytool -keystore keystore -alias jetty -genkey -keyalg RSA
# 如下输入
Enter keystore password:  			# 666666   密码可以自定义
Re-enter new password: 				# 666666	 重复密码
What is your first and last name?		# 回车
  [Unknown]:  
What is the name of your organizational unit?	# 回车
  [Unknown]:  
What is the name of your organization?		# 回车
  [Unknown]:  
What is the name of your City or Locality?	# 回车
  [Unknown]:  
What is the name of your State or Province?	# 回车
  [Unknown]:  
What is the two-letter country code for this unit?	# 回车
  [Unknown]:  
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?		# yes

Enter key password for <jetty>
	(RETURN if same as keystore password):  	# 回车 

4.1.2 修改web-server的config配置文件

[root@node3 server]# cd /usr/local/azkaban/server/conf
[root@node3 conf]# ls
azkaban.properties  azkaban-users.xml
[root@node3 conf]# vim azkaban.properties

# 修改azkaban.properties配置

#Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai  #改成上海

#Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml

#Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects

database.type=mysql
mysql.port=3306
mysql.host=XXX.XXX.XXX.XXX     #改成本机IP地址（公网）
mysql.database=azkaban        #已创建对应数据库
mysql.user=root                       # mysql用户名
mysql.password=123456           # mysql登录密码     
mysql.numconnections=100

# Velocity dev mode
velocity.dev.mode=false

# Azkaban Jetty server properties.
jetty.maxThreads=25
jetty.ssl.port=8443      
jetty.port=8081
jetty.keystore=keystore
jetty.password=666666    #对应设置的jetty密码
jetty.keypassword=666666
jetty.truststore=keystore
jetty.trustpassword=666666

# Azkaban Executor settings
executor.port=12321

# mail settings
mail.sender=
mail.host=
job.failure.email=
job.success.email=

lockdown.create.projects=false

cache.directory=cache
 

[root@node3 conf]# vi azkaban-users.xml 

# 修改azkaban-users配置，即用户界面登录时候的默认用户名

<azkaban-users>
        <user username="azkaban" password="azkaban" roles="admin" groups="azkaban" />
        <user username="metrics" password="metrics" roles="metrics"/>
        <user username="name" password="password" roles="admin,metrics"/>

#增加一行 用户自定义的用户名和密码

        <role name="admin" permissions="ADMIN" />
        <role name="metrics" permissions="METRICS"/>
</azkaban-users>
 

4.2 更新executor配置文件

[root@node3 conf]# cd /usr/local/azkaban/executor/conf
[root@node3 conf]# ls
azkaban.private.properties  azkaban.properties  global.properties

[root@node3 conf]# vi azkaban.properties

# 修改azkaban属性配置项 

#Azkaban
default.timezone.id=Asia/Shanghai  #同理修改为+8时区

# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=plugins/jobtypes

#Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects

database.type=mysql
mysql.port=3306
mysql.host=XXX.XXX.XXX.XXX  #mysql主机IP
mysql.database=azkaban
mysql.user=root           #用户名
mysql.password=123456          #密码
mysql.numconnections=100

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
 

5.启动和报错处理

5.1 启动web-server服务

[root@node3 conf]# cd /usr/local/azkaban/server
[root@node3 server]# cd bin
[root@node3 bin]# ./azkaban-web-start.sh 
Using Hadoop from /usr/local/hadoop
Using Hive from /usr/local/hive
./..
.:/usr/local/jdk/jvm/lib/dt.jar:/usr/local/jdk/jvm/lib/tools.jar:./../lib/azkaban-2.5.0.jar:./../lib/commons-collections-3.2.1.jar:./../lib/commons-configuration-1.8.jar:./../lib/commons-dbcp-1.4.jar:./../lib/commons-dbutils-1.5.jar:./../lib/commons-email-1.2.jar:./../lib/commons-fileupload-1.2.1.jar:./../lib/commons-io-2.4.jar:./../lib/commons-jexl-2.1.1.jar:./../lib/commons-lang-2.6.jar:./../lib/commons-logging-1.1.1.jar:./../lib/commons-pool-1.6.jar:./../lib/guava-13.0.1.jar:./../lib/h2-1.3.170.jar:./../lib/httpclient-4.2.1.jar:./../lib/httpcore-4.2.1.jar:./../lib/jackson-core-asl-1.9.5.jar:./../lib/jackson-mapper-asl-1.9.5.jar:./../lib/jetty-6.1.26.jar:./../lib/jetty-util-6.1.26.jar:./../lib/joda-time-2.0.jar:./../lib/jopt-simple-4.3.jar:./../lib/junit-4.8.jar:./../lib/log4j-1.2.16.jar:./../lib/mail-1.4.5.jar:./../lib/mysql-connector-java-8.0.28.jar:./../lib/servlet-api-2.5.jar:./../lib/slf4j-api-1.6.1.jar:./../lib/slf4j-log4j12-1.6.4.jar:./../lib/velocity-1.7.jar:./../lib/velocity-tools-2.0.jar:./../extlib/*.jar:./../plugins/*/*.jar:/usr/local/hadoop/conf:/usr/local/hadoop/*:/usr/local/hive/conf:/usr/local/hive/lib/*
[root@node3 bin]# 2022/05/30 00:23:24.183 +0800 ERROR [AzkabanWebServer] [Azkaban] Starting Jetty Azkaban Executor...
2022/05/30 00:23:24.201 +0800 INFO [AzkabanServer] [Azkaban] Loading azkaban settings file from ./../conf
2022/05/30 00:23:24.202 +0800 INFO [AzkabanServer] [Azkaban] Loading azkaban properties file
2022/05/30 00:23:24.214 +0800 INFO [AzkabanWebServer] [Azkaban] Setting up Jetty Https Server with port:8443 and numThreads:25
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/azkaban/server/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022/05/30 00:23:24.248 +0800 INFO [log] [Azkaban] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2022/05/30 00:23:24.273 +0800 INFO [AzkabanWebServer] [Azkaban] Loading user manager class azkaban.user.XmlUserManager
2022/05/30 00:23:24.273 +0800 ERROR [AzkabanWebServer] [Azkaban] Could not instantiate UserManager azkaban.user.XmlUserManager
Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at azkaban.webapp.AzkabanWebServer.loadUserManager(AzkabanWebServer.java:230)
	at azkaban.webapp.AzkabanWebServer.<init>(AzkabanWebServer.java:181)
	at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:726)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at azkaban.webapp.AzkabanWebServer.loadUserManager(AzkabanWebServer.java:226)
	... 2 more
Caused by: java.lang.IllegalArgumentException: User xml file conf/azkaban-users.xml doesn't exist.
	at azkaban.user.XmlUserManager.parseXMLFile(XmlUserManager.java:87)
	at azkaban.user.XmlUserManager.<init>(XmlUserManager.java:81)
	... 7 more

报错 ，原因：

azkaban-web-start.sh中默认路径根目录为 /usr/local/azkaban/server

切换到 /usr/local/azkaban/server/bin目录下启动脚本会找不到conf/azkaban-users.xml 文件和azkaban.user.XmlUserManager实例

解决办法：

启动azkaban-web-start.sh时 在server目录下运行 ./bin/azkaban-web-start.sh启动

 报错2： MySQL connection test failed.

2022/05/29 23:10:16.056 +0800 ERROR [DataSourceUtils] [Azkaban] MySQL connection test failed. Please check MySQL connection health!

原因：web-server模块中jdbc版本和mysql版本不一致导致的。

解决办法：替换jdbcJAR包，重启服务。

[root@node3 server]# cd  /usr/local/azkaban/server/lib

[root@node3 lib]# cp ~/mysql-connector-java-8.0.28.jar ./

[root@node3 lib]# ll
total 14376
-rw-r--r-- 1 root root  626798 Apr 22  2014 azkaban-2.5.0.jar
-rw-r--r-- 1 root root  575389 Apr 22  2014 commons-collections-3.2.1.jar
-rw-r--r-- 1 root root  354491 Apr 22  2014 commons-configuration-1.8.jar
-rw-r--r-- 1 root root  160519 Apr 22  2014 commons-dbcp-1.4.jar
-rw-r--r-- 1 root root   61898 Apr 22  2014 commons-dbutils-1.5.jar
-rw-r--r-- 1 root root   32321 Apr 22  2014 commons-email-1.2.jar
-rw-r--r-- 1 root root   57779 Apr 22  2014 commons-fileupload-1.2.1.jar
-rw-r--r-- 1 root root  185140 Apr 22  2014 commons-io-2.4.jar
-rw-r--r-- 1 root root  267634 Apr 22  2014 commons-jexl-2.1.1.jar
-rw-r--r-- 1 root root  284220 Apr 22  2014 commons-lang-2.6.jar
-rw-r--r-- 1 root root   60686 Apr 22  2014 commons-logging-1.1.1.jar
-rw-r--r-- 1 root root  111119 Apr 22  2014 commons-pool-1.6.jar
-rw-r--r-- 1 root root 1891110 Apr 22  2014 guava-13.0.1.jar
-rw-r--r-- 1 root root 1469903 Apr 22  2014 h2-1.3.170.jar
-rw-r--r-- 1 root root  427021 Apr 22  2014 httpclient-4.2.1.jar
-rw-r--r-- 1 root root  223374 Apr 22  2014 httpcore-4.2.1.jar
-rw-r--r-- 1 root root  228417 Apr 22  2014 jackson-core-asl-1.9.5.jar
-rw-r--r-- 1 root root  778983 Apr 22  2014 jackson-mapper-asl-1.9.5.jar
-rw-r--r-- 1 root root  539912 Apr 22  2014 jetty-6.1.26.jar
-rw-r--r-- 1 root root  177131 Apr 22  2014 jetty-util-6.1.26.jar
-rw-r--r-- 1 root root  569231 Apr 22  2014 joda-time-2.0.jar
-rw-r--r-- 1 root root   54584 Apr 22  2014 jopt-simple-4.3.jar
-rw-r--r-- 1 root root  236674 Apr 22  2014 junit-4.8.jar
-rw-r--r-- 1 root root  481535 Apr 22  2014 log4j-1.2.16.jar
-rw-r--r-- 1 root root  508143 Apr 22  2014 mail-1.4.5.jar
-rw-r--r-- 1 root root  875336 Apr 22  2014 mysql-connector-java-5.1.28.jar
-rwxr--r-- 1 root root 2476480 May 29 23:14 mysql-connector-java-8.0.28.jar
-rw-r--r-- 1 root root  105112 Apr 22  2014 servlet-api-2.5.jar
-rw-r--r-- 1 root root   25496 Apr 22  2014 slf4j-api-1.6.1.jar
-rw-r--r-- 1 root root    9748 Apr 22  2014 slf4j-log4j12-1.6.4.jar
-rw-r--r-- 1 root root  449505 Apr 22  2014 velocity-1.7.jar
-rw-r--r-- 1 root root  346580 Apr 22  2014 velocity-tools-2.0.jar

[root@node3 lib]# mv mysql-connector-java-5.1.28.jar mysql-connector-java-5.1.28.jar.back

# 重命名使之失效

同理，executor程序也可能有JDBC版本不一致的原因，一样修改。 

报错3：plugin path plugins/triggers doesn't exist!

2022/05/30 00:41:02.745 +0800 ERROR [AzkabanWebServer] [Azkaban] plugin path plugins/triggers doesn't exist!

原因：executor程序未启动。

报错4：Invalid maximum heap size: -Xmx4G

原因：用户虚拟内存过大。

处理方法：修改 azkaban-web-start.sh 文件，-Xmx4G修改成 -Xmx512M

修改 azkaban-executor-start.sh 文件，-Xmx3G修改成 -Xmx512M

5.2 启动executor服务

报错1：

org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not create connection to database server.)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at azkaban.database.DataSourceUtils$MySQLBasicDataSource$MonitorThread.pingDB(DataSourceUtils.java:186)
	at azkaban.database.DataSourceUtils$MySQLBasicDataSource$MonitorThread.run(DataSourceUtils.java:174)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.

原因：同5.1的报错2，jdbc的JAR包版本与mysql不匹配。替换即可。

报错2：

javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

原因：未知ssl根证书，可以忽略

若web-server 和 executor不再出现ERROR告警则azkaban启动成功。

运行jps如下：

[root@node3 server]# jps
2408 AzkabanWebServer
4329 Jps
2907 AzkabanExecutorServer

6.web登录

6.1 访问登录

浏览器访问https://hostip:8443/

输入用户名密码（azkaban-users.xml中配置项）登录

6.2 创建program工程

6.2.1 界面创建工作流

输入program name 和description 

6.2.2 创建任务文件，打包任务，上传 

创建任务，并且配置好前置（dependencies）后置（dependency）依赖

注意：command命令的值为纯shell指令，不用整个用引号。

#创建任务start.job

type=noop

# 创建任务111.job
type=command
command=echo 111
dependencies=start

# 创建任务222.job
type=command
command=echo 222
dependencies=start

# 创建任务333.job
type=command
command=echo 333
dependencies=start

# 创建任务444.job
type=command
command=echo 444
dependencies=start

#创建任务end.job

type=noop
dependencies=111,222,333,444

6.2.3 压缩，打包，上传。

任务文件压缩成zip格式

web 界面 upload

6.3 运行

 任务处理完毕，log可以查看详细日志。

end;