踩坑记——Azkaban Hdfs Plugin 安装

本文详细记录了:centos7.2, cdh 5.12 环境下azkaban 3.20.0 Hdfs插件的安装过程。因为官网文档在此部分的说明极为简陋,且3.x的插件需要源码构建并且问题诸多,因此留下此篇文章助诸位过坑。


官网文档:https://azkaban.github.io/azkaban/docs/latest/#hdfs-browser


一. Ant环境安装


1. 安装包下载

wget http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.10.1-bin.tar.gz


2. 解压至目录

tar zxvf apache-ant-1.10.1-bin.tar.gz 
mkdir /opt/ant
mv xxx /opt/ant

3. 添加环境变量并使其生效

vim /etc/profile
export ANT_HOME=/opt/ant
export PATH=${PATH}:$ANT_HOME/bin
source /etc/profile

4.  检查环境变量是否配置成功

ant -version

如图:




二. NodeJs安装


1. 安装包下载

curl --silent --location https://rpm.nodesource.com/setup_5.x | bash -

2. yum安装

yum install -y nodejs


三. JCE配置(默认JDK秘钥限制策略问题)


1. JCE下载(此处是jdk8的jce,请选择对应版本)

wget http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip

2. 解压并替换

替换目录:java_home/jre/lib/security/下的 local_policy.jar 和 US_export_policy.jar(注意备份)



四. 插件安装


现在正式开始插件的编译安装

1. 根据官网提示,首先 npm 安装 dustjs-linkedin

npm install -g less dustjs-linkedin

2. 下载插件源码并解压

地址:https://github.com/azkaban/azkaban-plugins


3. 进入解压后的根目录,执行ant编译


如上图所示,然后执行:

ant


4. 进入hadoopsecuritymanager目录,执行ant编译

hadoop 1.x:

cd plugins/hadoopsecuritymanager
ant

hadoop 2.x:

cd plugins/hadoopsecuritymanager-yarn
ant

编译结果位于 ../../dist 下对应路径的 jar目录下


5. 进入 hdfs 插件目录,执行ant编译

cd plugins/hdfsviewer
ant package


6. 将编译好的插件tar包,放入azkaban_web的插件目录下解压并修改目录名为hdfs

编译成功的插件包 xxx.tar.gz 位于 ../../dist/jobtype/packages/ 目录下

1) 创建viewer目录

mkdir plugins/viewer


2) 将tar包在viewer目录下解压并改名为hdfs

tar zxvf xxx.tar
mv xxxx hdfs



7. 修改插件的配置文件

vim conf/plugin.properties

内容示例:

viewer.name=HDFS
viewer.path=hdfs # 与path中的名字对应
viewer.order=1
viewer.hidden=false
viewer.external.classpaths=extlib/
viewer.servlet.class=azkaban.viewer.hdfs.HdfsBrowserServlet
hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0 # hadoop 1.x请选择 xxx_H_1_0, hadoop 2.x请选择 xxx_H_2_0
azkaban.should.proxy=false 
proxy.user=azkaban
proxy.keytab.location=
allow.group.proxy=false
file.max.lines=1000


8. 复制hadoop相关依赖包至lib目录

注:插件下的extlib不好使,建议以如下方式把相关依赖放入azkaban_web根目录下的extlib

cp /opt/cloudera/parcels/CDH/jars/commons-cli-1.2.jar hadoop-auth-2.5.1.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/hadoop-auth-2.6.0-cdh5.12.0.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/hadoop-common-2.6.0-cdh5.12.0.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/hadoop-hdfs-2.6.0-cdh5.12.0.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/protobuf-java-2.5.0.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/commons-configuration-1.7.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/hadoop-yarn-api-2.6.0-cdh5.12.0.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib
cp /opt/cloudera/parcels/CDH/jars/hadoop-yarn-common-2.6.0-cdh5.12.0.jar /opt/azkaban/azkaban-web-server-3.20.0/extlib


下述2个jar,因版本和依赖性较为特殊,可从maven单独下载并放入lib目录,在此提供pom信息:

<dependency>
<groupId>org.apache.htrace</groupId>
<artifactId>htrace-core</artifactId>
<version>4.1.0-incubating</version>
<dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-exec</artifactId>
    <version>2.1.0</version>
</dependency>

效果如图所示:


9. 复制azkaban hadoopsecuritymanager jar至插件lib目录

hadoop 1.x:

cp /opt/azkaban/azkaban-plugins-master/dist/hadoopsecuritymanager/jars/azkaban-hadoopsecuritymanager-3.0.0.jar /opt/azkaban/azkaban-web-server-3.20.0/plugins/viewer/hdfs/lib


hadoop 2.x:

cp /opt/azkaban/azkaban-plugins-master/dist/hadoopsecuritymanager-yarn/jars/azkaban-hadoopsecuritymanageryarn-3.0.0.jar /opt/azkaban/azkaban-web-server-3.20.0/plugins/viewer/hdfs/lib



10. 修改azkaban_web 的配置文件,添加插件路径

vim conf/azkaban.properties

内容:

# Azkaban plugin settings
azkaban.hdfsviewer.plugin.dir=plugins/viewer/hdfs/


11. 手动添加hadoop环境变量

vim /etc/profile
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29
export HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.yarn
source /etc/profile


12. 重启服务

./bin/azkaban-web-shutdown.sh
./bin/azkaban-web-start.sh



展开阅读全文

没有更多推荐了,返回首页