@羲凡——只为了更好的活着
Oozie编译安装完整版——Oozie4.3.1
Oozie是大数据生态圈中任务调度工具，对于一些初创公司来说，是很简单上手且节省开发成本，对于成熟的大公司来说，一般会自己开发一套调度系统。下面来编译安装Oozie4.3.1
一、编译
0.环境准备
下面是官网要求
System Requirements
Unix box (tested on Mac OS X and Linux)
Java JDK 1.7+
Maven 3.0.1+
Hadoop 0.20.2+
Pig 0.7+
JDK commands (java, javac) must be in the command path.
The Maven command (mvn) must be in the command path.
分别运行java -version、mvn -version、hadoop version、pig -version命令查看jdk、maven、hadoop、pig对应版本。显示如下
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /usr/local/package/maven-3.5.4
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/local/package/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-91-generic", arch: "amd64", family: "unix"
Hadoop 2.7.3
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff
Compiled by root on 2016-08-18T01:41Z
Compiled with protoc 2.5.0
Apache Pig version 0.17.0 (r1797386)
compiled Jun 02 2017, 15:41:58
1.下载oozie源码包
下载地址 https://archive.apache.org/dist/oozie/4.3.1/
2.解压
tar -zxf software/oozie-4.3.1.tar.gz -C ./
3.修改oozie-4.3.1/pom.xml文件
将把各个参数调整为自己公司组件的版本
<targetJavaVersion>1.8</targetJavaVersion>
<sourceJavaVersion>1.8</sourceJavaVersion>
<minJavaVersion>1.8</minJavaVersion>
<hadoop.version>2.7.3</hadoop.version>
<hive.version>1.2.2</hive.version>
<spark.version>2.3.2</spark.version>
<spark.streaming.kafka.version>2.3.2</spark.streaming.kafka.version>
<spark.bagel.version>2.3.2</spark.bagel.version>
<spark.scala.binary.version>2.11</spark.scala.binary.version>
4.执行编译命令
bin/mkdistro.sh -DskipTests
5.查看编译结果
如果出现如下内容，则说明你已经编译成功
编译成功的文件落在 oozie-4.3.1/distro/target/ 目录下，文件为 oozie-4.3.1-distro.tar.gz
二、安装
0.添加参数到Hadoop的core-site.xml文件中
其中 deplab 是你安装oozie的用户名
<property>
<name>hadoop.proxyuser.deplab.hosts</name>
<!--name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name-->
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.deplab.groups</name>
<!--name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name-->
<value>*</value>
</property>
1.解压并配置/etc/profile
tar -zxf software/oozie-4.3.1-distro.tar.gz -C ./
sudo vi /etc/profile
#oozie
export OOZIE_HOME=/usr/local/package/oozie-4.3.1
export PATH=$PATH:$OOZIE_HOME/bin
source /etc/profile
2.解压三个tar包
cd oozie-4.3.1/
tar -zxf oozie-client-4.3.1.tar.gz
tar -zxf oozie-examples.tar.gz
tar -zxf oozie-sharelib-4.3.1.tar.gz
3.修改conf/oozie-site.xml
<configuration>
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/usr/local/package/hadoop-2.7.3/etc/hadoop/</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.action.configurations</name>
<value>*=/usr/local/package/hadoop-2.7.3/etc/hadoop/</value>
</property>
<property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>*=/usr/local/package/spark-2.3.2-bin-hadoop2.7/conf</value>
</property>
<property>
<name>oozie.db.schema.name</name>
<value>oozie</value>
<description>Oozie DataBase Name</description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
<description>JDBC driver class.</description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://deptest37:3306/${oozie.db.schema.name}?createDatabaseIfNotExist=true&useSSL=false</value>
<description> JDBC URL. </description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>root</value>
<description>DB user name.</description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.password</name>
<value>1q2w3e4r</value>
<description>DB user password.</description>
</property>
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/${user.name}/share/lib</value>
<description>
System library path to use for workflow applications.
This path is added to workflow application if their job properties sets
the property 'oozie.use.system.libpath' to true.
</description>
</property>
<property>
<name>oozie.processing.timezone</name>
<value>GMT+0800</value>
</property>
</configuration>
特别检查 oozie.service.JPAService.jdbc.url 参数，下图右边圈出来的地方保持一致
4.生成libetx文件夹
mkdir libext
wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
cp ext-2.2.zip libext/
cp ${HADOOP_HOME}/share/hadoop/*/*.jar libext/
cp ${HADOOP_HOME}/share/hadoop/*/lib/*.jar libext/
5.生成sharelib文件夹
oozie-setup.sh sharelib create -fs hdfs://ns -locallib oozie-sharelib-4.3.1.tar.gz
会在hdfs上生成 /user/deplab/share/lib/lib_20190124163230文件
6.配置mysql元数据库(默认你已经安装mysql库)
下载mysql驱动并拷贝到libext和lib文件夹下
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.44/mysql-connector-java-5.1.44.jar
cp mysql-connector-java-5.1.44.jar libext/
cp mysql-connector-java-5.1.44.jar lib/
生成数据库
bin/ooziedb.sh create -sqlfile oozie.sql
bin/oozie-setup.sh db create -run -sqlfile oozie.sql
6.打war包
oozie-setup.sh prepare-war
当显示 INFO: Oozie is ready to be started表示war包成功
7.启动
oozie的启动需要开启Hadoop的 JobHistoryServer 服务
mr-jobhistory-daemon.sh start historyserver
oozie启动
oozied.sh start
输入一下命令检查oozie状态，如果显示 System mode: NORMAL者表示oozie启动正常
oozie admin -oozie http://deptest45:11000/oozie -status
oozie关闭命令是oozied.sh stop，如果这个命令停不掉，可以直接删除oozie开启是存放pid的文件
rm -rf /usr/local/package/oozie-4.3.1/oozie-server/temp/oozie.pid
8.修改oozie时区
a.oozie-site.xml 中的 oozie.processing.timezone 一定要先修改成 GMT+0800
b. vi oozie-server/webapps/oozie/oozie-console.js 将 GMT 改成 GMT+0800
function getTimeZone() {
Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
return Ext.state.Manager.get("TimezoneId","GMT+0800");
}
如果之前已经开启过ooozie的需要清除浏览器历史缓存(用谷歌浏览器)
三、报错处理
1.启动后任务不能运行，报错如下
java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.timeline.TimelineUtils.buildTimelineTokenService(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/io/Text;
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:171)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapred.ResourceMgrDelegate.<init>(ResourceMgrDelegate.java:96)
at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
at org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:514)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:512)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:512)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1429)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1142)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1386)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
解决方案：关闭oozie，执行下面命令，在启动oozie后，将 oozie-server/webapps/oozie/WEB-INF/lib 目录下的所有有关hadoop2.4.0的jar包全部删除即可
jar -xvf oozie.war
rm -rf oozie.war/WEB-INF/lib/hadoop-*.jar
jar -cvf oozie.war ./*
oozie-setup.sh prepare-war
2.报错mysql连接不上
a.先检查你的mysql的配置有没有问题，比如bind-address 有没有设置成 0.0.0.0
b.检查你的mysql驱动有没有拷贝到libext和lib中
c.有时需要将mysql驱动拷贝拷贝到oozie-server/lib目录下(一般不用)
3.报错缺少包
Caused by: java.lang.UnsatisfiedLinkError: /usr/local/package/jdk1.8.0_131/jre/lib/amd64/libawt_xawt.so: libXrender.so.1: cannot open shared object file: No such file or directory
解决方案：
sudo apt-get install libXrender-dev
sudo apt-get install libxtst-dev
====================================================================
@羲凡——只为了更好的活着
若对博客中有任何问题，欢迎留言交流
扫一扫
370
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
