Oozie编译安装完整版——Oozie4.3.1

@羲凡——只为了更好的活着

Oozie编译安装完整版——Oozie4.3.1

Oozie是大数据生态圈中任务调度工具,对于一些初创公司来说,是很简单上手且节省开发成本,对于成熟的大公司来说,一般会自己开发一套调度系统。下面来编译安装Oozie4.3.1

一、编译

0.环境准备

下面是官网要求

System Requirements
Unix box (tested on Mac OS X and Linux)
Java JDK 1.7+
Maven 3.0.1+
Hadoop 0.20.2+
Pig 0.7+
JDK commands (java, javac) must be in the command path.
The Maven command (mvn) must be in the command path.

分别运行java -versionmvn -versionhadoop versionpig -version命令查看jdk、maven、hadoop、pig对应版本。显示如下

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /usr/local/package/maven-3.5.4
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/local/package/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-91-generic", arch: "amd64", family: "unix"
Hadoop 2.7.3
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff
Compiled by root on 2016-08-18T01:41Z
Compiled with protoc 2.5.0
Apache Pig version 0.17.0 (r1797386) 
compiled Jun 02 2017, 15:41:58
1.下载oozie源码包

下载地址 https://archive.apache.org/dist/oozie/4.3.1/

2.解压
tar -zxf software/oozie-4.3.1.tar.gz -C ./
3.修改oozie-4.3.1/pom.xml文件

将把各个参数调整为自己公司组件的版本

<targetJavaVersion>1.8</targetJavaVersion>
<sourceJavaVersion>1.8</sourceJavaVersion>
<minJavaVersion>1.8</minJavaVersion>
<hadoop.version>2.7.3</hadoop.version>
<hive.version>1.2.2</hive.version>
<spark.version>2.3.2</spark.version>
<spark.streaming.kafka.version>2.3.2</spark.streaming.kafka.version>
<spark.bagel.version>2.3.2</spark.bagel.version>
<spark.scala.binary.version>2.11</spark.scala.binary.version>
4.执行编译命令
bin/mkdistro.sh -DskipTests
5.查看编译结果

如果出现如下内容,则说明你已经编译成功
在这里插入图片描述

编译成功的文件落在 oozie-4.3.1/distro/target/ 目录下,文件为 oozie-4.3.1-distro.tar.gz

二、安装

0.添加参数到Hadoop的core-site.xml文件中

其中 deplab 是你安装oozie的用户名

<property>
	<name>hadoop.proxyuser.deplab.hosts</name>
	<!--name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name-->
	<value>*</value>
</property>
<property>
	<name>hadoop.proxyuser.deplab.groups</name>
	<!--name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name-->
	<value>*</value>
</property>
1.解压并配置/etc/profile
tar -zxf software/oozie-4.3.1-distro.tar.gz -C ./
sudo vi /etc/profile
	#oozie
	export OOZIE_HOME=/usr/local/package/oozie-4.3.1
	export PATH=$PATH:$OOZIE_HOME/bin
source /etc/profile
2.解压三个tar包
cd oozie-4.3.1/
tar -zxf oozie-client-4.3.1.tar.gz
tar -zxf oozie-examples.tar.gz
tar -zxf oozie-sharelib-4.3.1.tar.gz
3.修改conf/oozie-site.xml
<configuration>
<property>  
	<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>  
	<value>*=/usr/local/package/hadoop-2.7.3/etc/hadoop/</value>  
</property>  
<property>  
	<name>oozie.service.HadoopAccessorService.action.configurations</name>  
	<value>*=/usr/local/package/hadoop-2.7.3/etc/hadoop/</value>  
</property>
<property>
	<name>oozie.service.SparkConfigurationService.spark.configurations</name>
	<value>*=/usr/local/package/spark-2.3.2-bin-hadoop2.7/conf</value>
</property>
<property>
	<name>oozie.db.schema.name</name>
	<value>oozie</value>
	<description>Oozie DataBase Name</description>
</property>
<property>
	<name>oozie.service.JPAService.jdbc.driver</name>
	<value>com.mysql.jdbc.Driver</value>
	<description>JDBC driver class.</description>
</property>
<property>
	<name>oozie.service.JPAService.jdbc.url</name>
	<value>jdbc:mysql://deptest37:3306/${oozie.db.schema.name}?createDatabaseIfNotExist=true&amp;useSSL=false</value>
	<description> JDBC URL. </description>
</property>
<property>
	<name>oozie.service.JPAService.jdbc.username</name>
	<value>root</value>
	<description>DB user name.</description>
</property>
<property>
	<name>oozie.service.JPAService.jdbc.password</name>
	<value>1q2w3e4r</value>
	<description>DB user password.</description>
</property>
<property>
	<name>oozie.service.WorkflowAppService.system.libpath</name>
	<value>/user/${user.name}/share/lib</value>
	<description>
	System library path to use for workflow applications.
	This path is added to workflow application if their job properties sets
	the property 'oozie.use.system.libpath' to true.
	</description>
</property>
<property>
	<name>oozie.processing.timezone</name>
	<value>GMT+0800</value>
</property>
</configuration>

特别检查 oozie.service.JPAService.jdbc.url 参数,下图右边圈出来的地方保持一致
在这里插入图片描述

4.生成libetx文件夹
mkdir libext 
wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
cp ext-2.2.zip libext/
cp ${HADOOP_HOME}/share/hadoop/*/*.jar libext/ 
cp ${HADOOP_HOME}/share/hadoop/*/lib/*.jar libext/ 
5.生成sharelib文件夹
oozie-setup.sh sharelib create -fs hdfs://ns -locallib oozie-sharelib-4.3.1.tar.gz

会在hdfs上生成 /user/deplab/share/lib/lib_20190124163230文件

6.配置mysql元数据库(默认你已经安装mysql库)

下载mysql驱动并拷贝到libext和lib文件夹下

wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.44/mysql-connector-java-5.1.44.jar
cp mysql-connector-java-5.1.44.jar  libext/
cp mysql-connector-java-5.1.44.jar  lib/

生成数据库

bin/ooziedb.sh create -sqlfile oozie.sql
bin/oozie-setup.sh db create -run  -sqlfile oozie.sql
6.打war包
oozie-setup.sh prepare-war

当显示 INFO: Oozie is ready to be started表示war包成功

7.启动

oozie的启动需要开启Hadoop的 JobHistoryServer 服务

mr-jobhistory-daemon.sh start historyserver

oozie启动

oozied.sh start

输入一下命令检查oozie状态,如果显示 System mode: NORMAL者表示oozie启动正常

oozie admin -oozie http://deptest45:11000/oozie -status

oozie关闭命令是oozied.sh stop,如果这个命令停不掉,可以直接删除oozie开启是存放pid的文件

rm -rf /usr/local/package/oozie-4.3.1/oozie-server/temp/oozie.pid
8.修改oozie时区

a.oozie-site.xml 中的 oozie.processing.timezone 一定要先修改成 GMT+0800
b. vi oozie-server/webapps/oozie/oozie-console.jsGMT 改成 GMT+0800

function getTimeZone() {
    Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
    return Ext.state.Manager.get("TimezoneId","GMT+0800");
}

如果之前已经开启过ooozie的需要清除浏览器历史缓存(用谷歌浏览器)

三、报错处理

1.启动后任务不能运行,报错如下
java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.timeline.TimelineUtils.buildTimelineTokenService(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/io/Text;
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:171)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.<init>(ResourceMgrDelegate.java:96)
	at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
	at org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:514)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:512)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:512)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1429)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1142)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1386)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
	at org.apache.oozie.command.XCommand.call(XCommand.java:287)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

解决方案:关闭oozie,执行下面命令,在启动oozie后,将 oozie-server/webapps/oozie/WEB-INF/lib 目录下的所有有关hadoop2.4.0的jar包全部删除即可

jar -xvf oozie.war
rm -rf oozie.war/WEB-INF/lib/hadoop-*.jar
jar -cvf oozie.war ./*
oozie-setup.sh prepare-war
2.报错mysql连接不上

a.先检查你的mysql的配置有没有问题,比如bind-address 有没有设置成 0.0.0.0
b.检查你的mysql驱动有没有拷贝到libext和lib中
c.有时需要将mysql驱动拷贝拷贝到oozie-server/lib目录下(一般不用)

3.报错缺少包
Caused by: java.lang.UnsatisfiedLinkError: /usr/local/package/jdk1.8.0_131/jre/lib/amd64/libawt_xawt.so: libXrender.so.1: cannot open shared object file: No such file or directory

解决方案:

sudo apt-get install libXrender-dev
sudo apt-get install libxtst-dev

====================================================================

@羲凡——只为了更好的活着

若对博客中有任何问题,欢迎留言交流

  • 5
    点赞
  • 11
    收藏
    觉得还不错? 一键收藏
  • 2
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值