Azkaban 3.90.0 高可用集群搭建

tony_wzx

已于 2024-05-27 13:25:19 修改

阅读量1.1k

点赞数 2

分类专栏： Linux 文章标签： linux 运维 devops 经验分享 segmentfault

于 2021-01-12 19:21:32 首次发布

本文链接：https://blog.csdn.net/tony_wzx/article/details/112544782

版权

Linux 专栏收录该内容

14 篇文章 1 订阅

订阅专栏

Azkaban的两种模式

solo-server模式：exec进程和web进程为同一个进程，存放元数据的数据库为H2（自带数据库）
multiple-executor模式：exec进程和web进程在不同的机器上，存放元数据的数据库为mysql

条件允许的话，可以剥离mysql，并添加多个azkaban-exec-server

服务器	组件	数据库
172.16.0.102	azkaban-web-server	mysql-server-slave
172.16.0.103	azkaban-exec-server	mysql-server-master

一、部署

1、172.16.0.103
1、部署 mysql5.7 或者5.6 （部署mysql8 会有异常，请看文章末尾）

#创建并使用数据库
CREATE DATABASE azkaban;
use azkaban;

#创建用户并授权
CREATE USER 'azkaban'@'%' IDENTIFIED BY '*****'; 
GRANT ALL ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

修改配置

[mysqld]
...
max_allowed_packet=1024M

2、部署 jdk8 略，设置环境变量

3、下载安装包，开始编译

yum install -y  git gcc-c++
 
cd /opt/src
 
wget https://github.com/azkaban/azkaban/archive/3.90.0.tar.gz
tar xf azkaban-3.90.0.tar.gz
 
cd azkaban-3.90.0
#编译 第一次运行时，此过程时间会比较长
#-x test 表示跳过测试
./gradlew build installDist -x test
#编译完成看查看tar包

ll azkaban-*/build/distributions/*.tar.gz
[root@172.16.0.103:azkaban-3.90.0]# ll azkaban-*/build/distributions/*.tar.gz
-rw-r--r-- 1 root root 5925 Dec 30 11:32 azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 16238767 Dec 30 11:55 azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 23139 Dec 30 11:46 azkaban-hadoop-security-plugin/build/distributions/azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 36379429 Dec 30 11:56 azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 20304106 Dec 30 11:42 azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz

把编译好的tar包拷贝出来，后面要用的就是这些

cp azkaban-*/build/distributions/*.tar.gz ./

组件	功能
azkaban-common	常用工具类
azkaban-db	对应的sql脚本
azkaban-exec-server	azkaban的executor-server单独模块
azkaban-hadoop-secutity-plugin	hadoop有关kerberos插件
azkaban-solo-server	web和executor运行在同一进程的项目
azkaban-spi	azkaban存储接口以及exception类
azkaban-web-server	azkaban的web-server单独模块

编译完成后，导入表结构（编译完成后，此文件就会出来，也可以去azkaban-db-0.1.0-SNAPSHOT.tar.gz里面找除了update开头的文件都导入，简单的方法就是导入下面的文件）

/opt/src/azkaban-3.90.0/azkaban-db/build/sql/create-all-sql-0.1.0-SNAPSHOT.sql

/opt/app/mysql5/bin/mysql -uroot -p******** azkaban < create-all-sql-0.1.0-SNAPSHOT.sql

mkdir /opt/app/azkaban && cd /opt/app/azkaban
tar xf  azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C /opt/app/azkaban
mv azkaban-exec-server-0.1.0-SNAPSHOT  azkaban-exec-server
cd azkaban-exec-server
 
[root@172.16.0.103:azkaban-exec-server-0.1.0-SNAPSHOT]# ll
total 16
drwxr-xr-x 3 root root 4096 Dec 30 11:55 bin
drwxr-xr-x 2 root root 4096 Dec 30 11:55 conf
drwxr-xr-x 2 root root 4096 Dec 30 11:55 lib
drwxr-xr-x 3 root root 4096 Dec 30 11:55 plugins

#编辑配置文件,修改参数
vim conf/azkaban.properties
default.timezone.id=Asia/Shanghai
azkaban.webserver.url=http://172.16.0.102:8081
mysql.port=3306
mysql.host=172.16.0.103
mysql.database=azkaban
mysql.user=azkaban
mysql.password=*******

启动（一定要在当前目录下启动）

./bin/start-exec.sh

启动成功后会在当前目录下生成executor.port 文件，没有就检查日志
还要激活，出现success即可

cd /opt/app/azkaban/azkaban-exec-server
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo
# {"status":"success"}

2、172.16.0.102
1、部署 mysql5.7 略，设置主从

2、部署 jdk8 略，设置环境变量

3、将103上编译好的包 azkaban-web-server-0.1.0-SNAPSHOT.tar.gz 拷贝到102上

mkdir /opt/app/azkaban && cd /opt/app/azkaban
tar xf  azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C /opt/app/azkaban
mv azkaban-web-server-0.1.0-SNAPSHOT  azkaban-web-server
cd azkaban-web-server
 
 
#编辑配置文件
vim conf/azkaban.properties
default.timezone.id=Asia/Shanghai
mysql.port=3306
mysql.host=172.16.0.103
mysql.database=azkaban
mysql.user=azkaban
mysql.password=*******
 
 
#编辑用户配置文件
cat conf/azkaban-users.xml
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>
  <user password="******" roles="admin,metrics" username="admin"/>       #新增此行，拥有admin 和 metrics 两个权限的超级管理账户
 
  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

启动（一定要在当前目录下启动）
./bin/start-web.sh

访问

http://localhost:8081/

问题：

假如安装的是mysql 8，那么启动 azkaban-exec-server 会报 mysql连接错误如下：

错误代码

2021/01/12 18:11:40.405 +0800 ERROR [MySQLDataSource] [main] [Azkaban] Failed to find write-enabled DB connection. Wait 15 seconds and retry. No.Attempt = 1
java.sql.SQLException: Cannot create PoolableConnectionFactory (java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long)
    at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:669)
    at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:544)
    at azkaban.db.MySQLDataSource.getConnection(MySQLDataSource.java:78)
    at org.apache.commons.dbutils.AbstractQueryRunner.prepareConnection(AbstractQueryRunner.java:175)
    at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:286)
    at azkaban.db.DatabaseOperator.query(DatabaseOperator.java:69)
    at azkaban.executor.ExecutorDao.fetchExecutor(ExecutorDao.java:63)
    at azkaban.executor.JdbcExecutorLoader.fetchExecutor(JdbcExecutorLoader.java:291)
    at azkaban.execapp.AzkabanExecutorServer.insertExecutorEntryIntoDB(AzkabanExecutorServer.java:304)
    at azkaban.execapp.AzkabanExecutorServer.start(AzkabanExecutorServer.java:256)
    at azkaban.execapp.AzkabanExecutorServer.launch(AzkabanExecutorServer.java:163)
    at azkaban.execapp.AzkabanExecutorServer.main(AzkabanExecutorServer.java:159)
Caused by: java.sql.SQLException: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1086)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
    at com.mysql.jdbc.ConnectionImpl.buildCollationMapping(ConnectionImpl.java:1074)
    at com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3600)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2541)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
    at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55)
    at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
    at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:115)
    at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665)
    ... 11 more
Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long
    at com.mysql.jdbc.ConnectionImpl.buildCollationMapping(ConnectionImpl.java:1019)
    ... 27 more

是因为自带的mysql连接驱动jar包版本过低无法连接mysql8，请更换 azkaban-exec-server/lib/mysql-connector-java-5.1.28.jar 这个jar包

更换为mysql-connector-java-5.1.44.jar 5.1.44版本以上都可以。下载地址：https://repo1.maven.org/maven2/mysql/mysql-connector-java/

注意此时还是不能连接会报错如下，需要将jdbc数据库连接参数后面追加 useSSL=false ，例：url: jdbc:mysql://localhost:3306/azkaban?characterEncoding=utf-8&useSSL=false , 需要更改源码重新编译，有个取巧的办法就是更改数据库配置 如：mysql.database=azkaban?useSSL=false

错误代码

2021/01/12 18:37:54.666 +0800  INFO [log] [main] [Azkaban] Started SocketConnector@0.0.0.0:36839
2021/01/12 18:37:54.769 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] Tue Jan 12 18:37:54 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021/01/12 18:37:54.872 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] Tue Jan 12 18:37:54 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021/01/12 18:37:54.882 +0800  INFO [AzkabanExecutorServer] [main] [Azkaban] This executor wasn't found in the DB. Adding self.
2021/01/12 18:37:54.892 +0800  INFO [AzkabanExecutorServer] [main] [Azkaban] Started Executor Server on 172.16.0.103:36839
2021/01/12 18:37:54.952 +0800  INFO [AzkabanExecutorServer] [main] [Azkaban] Setting timezone to Asia/Shanghai

还是推荐使用mysql5.7 ，同时如果非要使用mysql8还是进行编译为好。

问题二

任务无法执行，状态为preparing

web日志如下：

2021/01/15 17:34:00.591 +0800  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Successfully refreshed executor: iZuf6e5whu8d7d5pcwmaxtZ:18219 (id: 6), active=true with executor info : ExecutorInfo{remainingMemoryPercent=79.77320197676259, remainingMemoryInMB=6024, remainingFlowCapacity=30, numberOfAssignedFlows=0, lastDispatchedTime=0, cpuUsage=0.0}
2021/01/15 17:34:00.591 +0800  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Using dispatcher for execution id :27
2021/01/15 17:34:00.599 +0800  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Reached handleNoExecutorSelectedCase stage for exec 27 with error count 0
2021/01/15 17:34:00.600 +0800  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Using dispatcher for execution id :27
2021/01/15 17:34:00.600 +0800  INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Reached handleNoExecutorSelectedCase stage for exec 27 with error count 0

默认情况下任务只分配给executor内存大于6G的节点执行，假如没有内存大于6G的executor，任务会持续等待直到有为止（6x1024=6144M）

日志中executor内存信息为 remainingMemoryInMB=6024 ，所以不执行！

可以再web中修改配置文件 conf/azkaban.properties 去掉 MinimumFreeMemory

azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

改为

azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus

重启web 即可