安装 准备: oozie-3.3.2,oozie-4.0.0在执行oozie-setup.sh时失败,所以选取3.3.2版本 mysql hadoop-1.0.3 apache-6 mavn3(必须是3或者以上版本) ext-2.2.zip拷贝至oozie-3.3.2目录下 创建 oozie 组和用户 /usr/sbin/groupadd oozie /usr/sbin/useradd oozie -g oozie 环境变量: export JAVA_HOME=/usr/local/jrockit-jdk1.6.0_29 export JRE_HOME=$JAVA_HOME/jre export HADOOP_HOME=/usr/local/hadoop-1.0.3 export HIVE_HOME=/usr/local/hive-0.9.0 export CATALINA_HOME=/home/qiulp/apache-tomcat-6.0.37 export OOZIE_HOME=/home/qiulp/oozie-3.3.2 export M2_HOME=/home/qiulp/apache-maven-3.1.0 export MAVEN_HOME=/home/qiulp/apache-maven-3.1.0 及相应的path 编译: bin/mkdistro.sh -DskipTests 跳过测试,不跳过时build failed 编译好的目录: oozie-3.3.2_bak/distro/target 拷贝出oozie-3.3.2-distro.tar.gz,原oozie-3.3.2就可以删除掉了,解压oozie-3.3.2-distro.tar.gz 修改bin/addtowar.sh elif [ "${version}" = "0.20.200" ]; then 成 elif [ "${version}" = "1.0.3" ]; then 创建libext目录 将mysql-connector-java-5.0.3-bin.jar分别放路lib、libext和CATALINA_HOME/lib下 执行:./oozie-setup.sh -hadoop 1.0.3 ${HADOOP_HOME} -extjs ../ext-2.2.zip INFO: Adding extension: /home/hadoop/oozie-3.3.2/libext/mysql-connector-java-5.0.3-bin.jar Injecting following Hadoop JARs /home/hadoop/hadoop-1.0.3/hadoop-core-1.0.3.jar /home/hadoop/hadoop-1.0.3/lib/jackson-core-asl-1.8.8.jar /home/hadoop/hadoop-1.0.3/lib/jackson-mapper-asl-1.8.8.jar /home/hadoop/hadoop-1.0.3/lib/commons-configuration-1.6.jar New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/hadoop/oozie-3.3.2/oozie-server/webapps/oozie.war INFO: Oozie is ready to be started mysql库创建(摘抄): mysql> create database oozie; mysql> create user 'oozie' identifiedby 'oozie'; //下面这条命令十分重要,‘%’代表可以从任意host访问,即使oozie用户具有远程访问权限。 mysql>grant all privileges on oozie.* to 'oozie'@'%' with grant option; mysql> grant all privileges on oozie.* to 'oozie'@'localhost' with grant option; //下面这条命令是这是oozie用户,从任意host访问的密码皆是'123456' mysql> update mysql.user set password=password('123456') where User="oozie" and Host="%"; mysql> update mysql.user set password=password('123456') where User="oozie" and Host="localhost"; mysql> flush privileges; 修改conf/oozie-site.xml <property> <name>oozie.service.JPAService.create.db.schema</name> <value>true</value> <description> Creates Oozie DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. </description> </property> <property> <name>oozie.service.JPAService.jdbc.driver</name> <value>com.mysql.jdbc.Driver</value> <description> JDBC driver class. </description> </property> <property> <name>oozie.service.JPAService.jdbc.url</name> <value>jdbc:mysql://10.200.187.77:3306/oozie</value> <description> JDBC URL. </description> </property> <property> <name>oozie.service.JPAService.jdbc.username</name> <value>oozie</value> <description> DB user name. </description> </property> <property> <name>oozie.service.JPAService.jdbc.password</name> <value>123456</value> <description> DB user password. IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value, if empty Configuration assumes it is NULL. </description> </property> 执行: ./ooziedb.sh create -sqlfile oozie.sql -run 日志 setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" Validate DB Connection DONE Check DB schema does not exist DONE Check OOZIE_SYS table does not exist DONE Create SQL schema DONE Create OOZIE_SYS table DONE Oozie DB has been created for Oozie version '3.3.2' 同时形成了oozie.sql执行过得语句,此时mysql已经创建好了所需的基础表。 最后修改hadoop的core-site.xml,增加配置,并重启 <property> <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name> <value>[OOZIE_SERVER_HOSTNAME]</value> </property> <property> <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name> <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value> </property> 这个配置很是麻烦,这里解释一下: OOZIE_SERVER_USER是oozie server的user OOZIE_SERVER_HOSTNAME是oozie server的hostname USER_GROUPS_THAT_ALLOW_IMPERSONATION是client端提交任务的usergroup 我安装的hadoop oozie都在同一台机器上,user非别是hadoop和oozie,client在hadoop用户组里,所以USER_GROUPS_THAT_ALLOW_IMPERSONATION是hadoop。 如下: <property> <name>hadoop.proxyuser.oozie.hosts</name> <value>master</value> </property> <property> <name>hadoop.proxyuser.oozie.groups</name> <value>hadoop</value> </property> </configuration> 这里我测试了多种情况,hadoop、oozieserver、oozieclient在不同的机器上机会报: Error: E0902 : E0902: Exception occured: [User: XXX is not allowed to impersonate XXX] 我的理解:如上的hadoop core配置中,OOZIE_SERVER_USER是oozie server的user,因为配置在hadoop master机器上所以此用户必须是同一台机器的相同或者不同user上。 同样USER_GROUPS_THAT_ALLOW_IMPERSONATION必须是同一台机器的相同或者不同usergroup里。 不知道是否正确,但试下来同一台机器是可以的,机器不同就example跑不起来。 运行 ./oozied.sh run 打开url http://localhost:11000/oozie/ oozie client部署和测试: 解压后的oozie-3.3.2内部含有oozie-client-3.3.2.tar.gz,oozie-examples.tar.gz 分别解压。 上传测试数据至hdfs,hadoop fs -put hadoop fs -put examples examples (至hdfs当前用户下,即.../usr/user/example,注意测试client的用户需要与数据上传的用户相同,或者上传时指定数据的绝对路径,不然测试job获取不到数据) examples/apps/map-reduce/job.properties修改hadoop的相关内容 提示:examples/apps下有各种测试样例,一个目录一个样例,如hive sqoop pig 运行测试样例: oozie-client-3.3.2/bin/oozie job -oozie http://10.200.187.77:11000/oozie -config examples/apps/map-reduce/job.properties -run