Azkaban hdfs plugin 配置

最新推荐文章于 2024-08-15 09:52:19 发布

liumu243

最新推荐文章于 2024-08-15 09:52:19 发布

阅读量2.1k

点赞数

分类专栏： azkaban 文章标签： azkaban hdfs 插件配置

本文链接：https://blog.csdn.net/liumu243/article/details/81288884

版权

azkaban 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

相关服务azkaban-web-server

/opt/soft/azkaban/azkaban-web-server

创建plugins/viewer目录

bin  conf  currentpid  keystore  lib  plugins  temp  web

相关插件包 azkaban-hdfs-viewer-3.0.0.tar.gz

解压到azkaban-web-server/plugins/viewer 并重命名为hdfs

tar -xvf azkaban-hdfs-viewer-3.0.0.tar.gz
mv azkaban-hdfs-viewer-3.0.0 hdfs

将azkaban-hadoopsecuritymanageryarn-3.0.0.jar(azkaban-plugin编译之后) 拷贝到 azkaban-web-server/plugins/viewer/hdfs/lib 目录下:

修改:/opt/soft/azkaban_/azkaban-web-server/plugins/viewer/hdfs/conf/plugin.properties

viewer.name=HDFS
viewer.path=hdfs
viewer.order=1
viewer.hidden=false
viewer.external.classpaths=extlib/
viewer.servlet.class=azkaban.viewer.hdfs.HdfsBrowserServlet
hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0
azkaban.should.proxy=false
proxy.user=azkaban
proxy.keytab.location=
allow.group.proxy=false
file.max.lines=1000

创建extlib/文件内容如下

将相关的Hadoop(此环境使用的是CDH5下相关jar包)的包拷贝到extlib目录下

azkaban-web-server/extlib

-rw-r--r-- 1 root root   25648 Jul 30 17:46 azkaban-hadoopsecuritymanageryarn-3.0.0.jar
-rwxr-xr-x 1 root root   41123 Jul 30 17:46 commons-cli-1.2.jar
-rwxr-xr-x 1 root root  350332 Jul 30 17:46 commons-configuration-1.7.jar
lrwxrwxrwx 1 root root      76 Jul 30 17:46 hadoop-auth.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop/hadoop-auth.jar
lrwxrwxrwx 1 root root      78 Jul 30 17:46 hadoop-common.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop/hadoop-common.jar
lrwxrwxrwx 1 root root      81 Jul 30 17:46 hadoop-hdfs.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop-hdfs/hadoop-hdfs.jar
lrwxrwxrwx 1 root root     105 Jul 30 17:46 hadoop-mapreduce-client-common.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar
lrwxrwxrwx 1 root root     103 Jul 30 17:46 hadoop-mapreduce-client-core.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar
lrwxrwxrwx 1 root root      85 Jul 30 17:46 hadoop-yarn-api.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop-yarn/hadoop-yarn-api.jar
lrwxrwxrwx 1 root root      88 Jul 30 17:46 hadoop-yarn-common.jar -> /opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3/lib/hadoop-yarn/hadoop-yarn-common.jar
-rwxr-xr-x 1 root root 1485102 Jul 30 17:46 htrace-core4-4.0.1-incubating.jar
-rwxr-xr-x 1 root root  533455 Jul 30 17:46 protobuf-java-2.5.0.jar

修改vim conf/azkaban.properties

# Azkaban plugin settings
azkaban.hdfsviewer.plugin.dir=plugins/viewer/hdfs/

手动添加hadoop环境变量

vi /etc/profile
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.13.2-1.cdh5.13.2.p0.3
export HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.yarn
source /etc/profile

重新启动:azkaban-web-server

效果: