azkaban安装

一路阳光CCG

已于 2022-11-07 10:28:31 修改

阅读量469

点赞数

分类专栏： linux 文章标签：数据库服务器 linux

于 2022-11-06 13:53:44 首次发布

本文链接：https://blog.csdn.net/yangguangniubi/article/details/127714431

版权

linux 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

我是用的mysql是8版本的,发现最新的阿兹卡班的master是支持mysql8的,因此可以直接下载编译

源码地址:GitHub - azkaban/azkaban: Azkaban workflow manager.

下载之后需要修改编译源码

修改源码

MySQLDataSource.java

进入到azkaban-db/src/main/java/azkaban/db/目录下

修改MySQLDataSource.java,在50行添加时区,不然会报错

addConnectionProperty("serverTimezone","Asia/Shanghai");

Caused by: java.sql.SQLException: The server time zone value '�й���׼ʱ��' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the 'serverTimezone' configuration property) to use a more specifc time zone value if you want to utilize time zone support.

安装编译环境

yum install -y git
yum install -y gcc-c++

开始编译

[root@hadoop102 azkaban-master]# ./gradlew build installDist -x test


Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.0/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 4s
85 actionable tasks: 15 executed, 70 up-to-date


表示成功

####进入目录
[root@hadoop102 azkaban-master]# cd azkaban-exec-server/build/install/
[root@hadoop102 install]# ll
总用量 0
drwxr-xr-x. 6 root root 55 11月  7 09:18 azkaban-exec-server
##########赋值到指定目录下
[root@hadoop102 install]# cp -r azkaban-exec-server/ /opt/module/azkaban/
进入
[root@hadoop102 install]# cd /opt/module/azkaban/
#########azkaban-exec-server修改名称为azkaban-exec

azkaban-exec

修改配置文件

[root@hadoop102 azkaban-exec]# vim conf/azkaban.properties 

default.timezone.id=Asia/Shanghai
azkaban.webserver.url=http://hadoop102:8081
###这里我改成了绝对路径
user.manager.xml.file=/opt/module/azkaban/azkaban-exec/conf/azkaban-users.xml
executor.global.properties=/opt/module/azkaban/azkaban-exec/conf/global.properties
azkaban.jobtype.plugin.dir=/opt/module/azkaban/azkaban-exec/plugins/jobtypes
azkaban.project.dir=/opt/module/azkaban/azkaban-exec/projects

database.type=mysql
mysql.port=3306
mysql.host=192.168.xxx.xx
mysql.database=azkaban
mysql.user=root
mysql.password=root
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30
executor.port=12321


#######修改完之后分发
[root@hadoop102 azkaban]# xsync azkaban-exec/

启动并激活

[root@hadoop102 azkaban-exec]# bin/start-exec.sh 
###有 executor.port说明启动成功,这里我用的是mysql8版本,所以我把jar包换了,大家不要忘记
[root@hadoop102 azkaban-exec]# ll
总用量 300
drwxr-xr-x. 3 root root     67 4月  18 2020 bin
drwxr-xr-x. 2 root root     81 11月  6 12:10 conf
-rw-r--r--. 1 root root      5 11月  6 12:04 currentpid
drwxr-sr-x. 2 root root      6 11月  6 12:04 executions
-rw-r--r--. 1 root root      5 11月  6 12:04 executor.port
-rw-r--r--. 1 root root 184161 11月  6 12:10 executorServerLog__2022-11-06+11:58:53.out
-rw-r--r--. 1 root root  17652 11月  6 12:03 executorServerLog__2022-11-06+12:01:23.out
-rw-r--r--. 1 root root   9528 11月  6 12:10 executorServerLog__2022-11-06+12:04:03.out
drwxr-xr-x. 2 root root   4096 11月  6 12:01 lib
drwxr-xr-x. 2 root root     36 11月  6 12:04 logs
drwxr-xr-x. 3 root root     22 4月  18 2020 plugins
drwxr-xr-x. 2 root root      6 11月  6 11:58 projects
drwxr-xr-x. 2 root root      6 11月  6 11:58 temp
#############激活
[root@hadoop102 azkaban-exec]# curl -G "hadoop102:12321/executor?action=activate" && echo
{"status":"success"}

azkaban-web

修改配置文件

azkaban.properties

[root@hadoop102 conf]# vim azkaban.properties

web.resource.dir=/opt/module/azkaban/azkaban-web/web/
default.timezone.id=Asia/Shanghai
user.manager.xml.file=/opt/module/azkaban/azkaban-web/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/opt/module/azkaban/azkaban-web/conf/global.properties
azkaban.project.dir=/opt/module/azkaban/azkaban-web/projects
database.type=mysql
mysql.port=3306
mysql.host=192.168.xxx.xx
mysql.database=azkaban
mysql.user=root
mysql.password=root
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus

azkaban-users.xml

添加一个zyg的用户,密码为bdy@2022,角色为admin

[root@hadoop102 conf]# vim azkaban-users.xml 

<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>
  <user password="bdy@2022" roles="admin" username="zyg"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

修改内存

4g改为1g,别问为什么,问就是因为穷

cd /opt/module/azkaban/azkaban-web/bin/internal
[root@hadoop102 internal]# vim internal-start-web.sh 

if [[ -z "$AZKABAN_OPTS" ]]; then
  AZKABAN_OPTS="-Xmx1G"
fi

启动


[root@hadoop102 azkaban-web]# bin/start-web.sh

报错

2022/11/07 10:07:24.077 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] SLF4J: Class path contains multiple SLF4J bindings.
2022/11/07 10:07:24.077 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] SLF4J: Found binding in [jar:file:/opt/module/azkaban/azkaban-web/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022/11/07 10:07:24.077 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] SLF4J: Found binding in [jar:file:/opt/module/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022/11/07 10:07:24.077 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022/11/07 10:07:24.079 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2022/11/07 10:07:24.172 +0800  INFO [AzkabanCommonModule] [main] [Azkaban] Offline logs loader is not enabled
2022/11/07 10:07:24.175 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.225 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.227 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.227 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.227 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.227 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.228 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.228 +0800  INFO [DispatchMethod] [main] [Azkaban] Value of dispatch method is : PUSH
2022/11/07 10:07:24.237 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] Exception in thread "main" 
2022/11/07 10:07:24.237 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at java.lang.Class.getDeclaredMethods0(Native Method)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at java.lang.Class.getDeclaredMethods(Class.java:1975)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:132)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.internal.ProviderMethodsModule.configure(ProviderMethodsModule.java:123)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:349)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.spi.Elements.getElements(Elements.java:110)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.Guice.createInjector(Guice.java:99)
2022/11/07 10:07:24.238 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.Guice.createInjector(Guice.java:73)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at com.google.inject.Guice.createInjector(Guice.java:62)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:255)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
2022/11/07 10:07:24.239 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	... 14 more

把jar包替换