Oozie-2-安装、配置、使用

一、Maven编译Oozie
tar -zxvf oozie-4.0.1.tar.gz
cd oozie-4.0.1
vi pom.xml


1.1、修改Java版本
<properties>
<javaVersion>1.6</javaVersion>
<targetJavaVersion>1.6</targetJavaVersion>


1.2、修改Hadoop版本
[root@bigdata3 oozie-4.0.1]# grep -l "2.3.0" `find . -name "pom.xml"`
./hadooplibs/hadoop-2/pom.xml
./hadooplibs/hadoop-distcp-2/pom.xml
./hadooplibs/hadoop-test-2/pom.xml
./pom.xml


/2.3.0
%s/2.3.0/2.0.6-alpha/g (%s/2.3.0/2.2.0/g)
/2.0.6-alpha


1.3、编译
mkdistro.sh -DskipTests -Dhadoop.version=2.0.6-alpha
编译后目的包的路径：$OOZIE_HOME
oozie-4.0.1/distro/target/oozie-4.0.1-distro.tar.gz


二、安装Oozie
2.1、解压编译后的包，mkdir libext，拷贝jar包
tar -zxvf oozie-4.1.0-distro.tar.gz
ln -s oozie-4.1.0 oozie


配置环境变量(vi /etc/profile)
#oozie410-3-16
export OOZIE_HOME=/opt/oozie410-3-16/oozie
export PATH=$PATH:$OOZIE_HOME/bin


#tomcat for oozie410-3-16
export TOMCAT_HOME=/opt/oozie410-3-16/tomcat
export CATALINA_HOME=/opt/oozie410-3-16/tomcat
export PATH=$PATH:$TOMCAT_HOME/bin:$CATALINA_HOME/bin


cd oozie
mkdir libext
cp /opt/oozie/oozie/libext/ext-2.2.zip ./libext/
cp /opt/hadoop/share/hadoop/*/*.jar ./libext/
cp /opt/hadoop/share/hadoop/*/lib/*.jar ./libext/
cp /opt/soft/apache-hive-0.13.1-bin/lib/mysql-connector-java-5.1.18-bin.jar ./libext/


删除/opt/oozie410/oozie/libext中多余的jar（在/opt/oozie410/oozie/oozie-server/lib中存在）：
rm -rf ./libext/jasper-runtime-5.5.23.jar;
rm -rf ./libext/jasper-compiler-5.5.23.jar;
rm -rf ./libext/servlet-api-2.5.jar;
rm -rf ./libext/jsp-api-2.1.jar


2.2、修改oozie-site.xml(vi ./conf/oozie-site.xml)
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>


<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://bigdata0:3306/oozie410</value>     


<name>oozie.service.JPAService.jdbc.username</name>
<value>root</value> 


<name>oozie.service.JPAService.jdbc.password</name>
<value>passw0rd</value>


<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/opt/hadoop/etc/hadoop</value>


【File /user/root/share/lib does not exist】
oozie-setup.sh sharelib create -fs hdfs://bigdata0:9000
oozie-setup sharelib create -fs FS_URI [-locallib SHARED_LIBRARY]
执行上述命令后，会生成/user/root/share/lib/lib_20141217115017
<!--不用指定，执行上面命令即可
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/root/share/lib/lib_20141216112609</value>
-->
生成Oozie所需要的数据表（初始化数据库）  
手动创建数据库oozie410，并进行访问授权
$OOZIE_HOME/bin/ooziedb.sh create -sqlfile oozie.sql -run
会在bin目录下生成oozie.sql文件


（./oozie-setup.sh db create -run  -sqlfile ./oozie.sqlhttp://michaellou.iteye.com/blog/2110420）


三、安装Oozie的war包
$OOZIE_HOME/bin/oozie-setup.sh prepare-war
会生成war包，路径：$OOZIE_HOME/oozie-server/webapps/oozie.war
Oozie自带的Tomcat路径：$OOZIE_HOME/oozie-server/ (/opt/oozie410/oozie/oozie-server/)


四、启动Oozie
oozie-run.sh前台运行
oozied.sh start后台运行


五、安装验证
5.1、浏览器查看
http://bigdata0:11000/oozie/
5.2、查看端口
netstat -ant | grep 1100
开启端口：11000、11001
5.3、检查Oozie的健康状况
oozie admin  -oozie http://bigdata0:11000/oozie/ -status
System mode: NORMAL


六、整合Oozie与Hadoop

修改core-site.xml，添加

<property>

     <name>hadoop.proxyuser.root.hosts</name>
     <value>bigdata0</value>
</property>
<property>
     <name>hadoop.proxyuser.root.groups</name>
     <value>root</value>
</property>


七、测试
7.1、运行mapreduce demo
hadoop fs -put /opt/oozie410/oozie/examples /user/root/examples
修改MapReduce例子，进入map-redice路径
cd /opt/oozie410/oozie/examples/apps/map-reduce
cd /home/jianxin/oozie/mapreduce


运行mapreduce任务
cd /opt/oozie410/oozie/oozie-client-4.1.0/bin
oozie job -oozie http://bigdata0:11000/oozie/ -config /opt/oozie410/oozie/examples/apps/map-reduce/job.properties -run
oozie job -oozie http://bigdata0:11000/oozie/ -config /home/jianxin/oozie/mapreduce/job.properties -run


oozie job -config /opt/oozie410/oozie/examples/apps/map-reduce/job.properties -run
检查任务状态
oozie job -oozie http://bigdata0:11000/oozie -info 0000001-141215165012581-oozie-root-W
oozie job -info 0000000-141216111330326-oozie-root-W

7.2、运行shell demo
修改任务参数
cd /opt/oozie410/oozie/examples/apps/shell
cd /home/jianxin/oozie/shell
执行客户端命令
cd /opt/oozie410/oozie/oozie-client-4.1.0/bin


oozie job -config /opt/oozie410/oozie/examples/apps/shell/job.properties -run
oozie job -config /home/jianxin/oozie/shell/job.properties -run
oozie job -info 0000001-141216111330326-oozie-root-W
7.3、Java
【Oozie demo目录】cd /opt/oozie410-3-16/oozie/examples/apps/java-main
【我自己的应用目录】cd /home/jianxin/oozie/java-main-demo
【运行job】oozie job -config /home/jianxin/oozie/java-main/job.properties -run
【查看job运行状况】oozie job -info 0000001-141216111330326-oozie-root-W


7.4、fs


7.5、Hive
oozie job -config /home/jianxin/oozie/hive-demo/job.properties -run


hadoop fs -rmr /oozie/mydemo/hive-demo/*;
hadoop fs -put * /oozie/mydemo/hive-demo/


八、其他命令
8.1、显示
oozie admin -oozie http://bigdata0:11000/oozie -shareliblist
oozie admin -oozie http://bigdata0:11000/oozie -sharelibupdate 
8.2、查看作业情况
oozie job -log 0000001-141219144911055-oozie-root-W
oozie job -configcontent 0000001-141219144911055-oozie-root-W
8.3、修改/etc/profile
export OOZIE_URL="http://localhost:11000/oozie"
8.4、校验workflow  XML是否可用

oozie validate ./workflow.xml