Install oozie-3.3.2 on Hadoop 1.1.1

After a few hours tweaking and googling, I managed to install apache oozie 3.3.2 on Hadoop 1.1.1.

The document provided in apache oozie 3.3.2 is not very clear. After some googling, I found this blog is a great guide for you install oozie 3.3.1 on Hadoop 0.23.0.

http://srikanthayalasomayajulu.blogspot.sg/2013/02/apache-oozie-331-installation-on-apache.html

If you want to install oozie on MRv2/YARN, please follow the blog post.

 

Here we go, we're going to install oozie 3.3.2 on Hadoop 1.1.1, which is offically supported version.

Apache oozie 3.3.2 official quick start page: http://oozie.apache.org/docs/3.3.2/DG_QuickStart.html

 

My local environment:

• Apache Hadoop 1.1.1 pseudo setup
• Apache oozie 3.3.2, the latest release
• Java 1.7 u25
• Apache Maven 3.0.4

Oozie server installation:

• • Download apache oozie 3.3.2 via: http://www.apache.org/dyn/closer.cgi/oozie/3.3.2
  • Unpack the oozie-3.3.2.tar.gz under development directory: ~/dev/oozie-3.3.2
  • The following two properties are required in hadoop core-site.xml:

  • <!-- OOZIE -->
      <property>
        <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
        <value>[OOZIE_SERVER_HOSTNAME]</value>
      </property>
      <property>
        <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
        <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
      </property>

     Replaced [OOZIE_SERVER_USER] with gsun , [OOZIE_SERVER_HOSTNAME] with * and [OOZIE_GROUPS_THAT_ALLOW_IMPERSONATION] with *  . TIP: in this blog post, the $OOZIE_HOME refers to ~/dev/oozie-3.3.2
  • Download extjs-2.2.zip, don't unzip it right now.
  • Building a distro

  • cd $OOZIE_HOME
    ./bin/mkdistro.sh -DskipTests

     It will take a while, go get a coffee, you'll be fine.
  • If the build successfully, we could go to the next step. In $OOZIE_HOME a directory called  *hadooplibs/* directory will be created containing the Hadoop JARs for the versions of Hadoop that the Oozie distribution supports.

    MacBookPro:oozie-3.3.2 gsun$ cd hadooplibs/
    MacBookPro:hadooplibs gsun$ ls
    hadoop-1/        hadoop-3/        hadoop-distcp-2/ hadoop-test-1/   hadoop-test-3/   target/
    hadoop-2/        hadoop-distcp-1/ hadoop-distcp-3/ hadoop-test-2/   pom.xml
    MacBookPro:hadooplibs gsun$ tree hadoop-1/
    hadoop-1/
    ├── pom.xml
    └── target
        ├── archive-tmp
        ├── hadooplibs
        │   └── hadooplib-1.1.1.oozie-3.3.2
        │       ├── commons-beanutils-1.7.0.jar
        │       ├── commons-beanutils-core-1.8.0.jar
        │       ├── commons-codec-1.4.jar
        │       ├── commons-collections-3.2.1.jar
        │       ├── commons-configuration-1.6.jar
        │       ├── commons-digester-1.8.jar
        │       ├── commons-el-1.0.jar
        │       ├── commons-io-2.1.jar
        │       ├── commons-lang-2.4.jar
        │       ├── commons-logging-1.1.jar
        │       ├── commons-math-2.1.jar
        │       ├── commons-net-1.4.1.jar
        │       ├── hadoop-client-1.1.1.jar
        │       ├── hadoop-core-1.1.1.jar
        │       ├── hsqldb-1.8.0.7.jar
        │       ├── jackson-core-asl-1.8.8.jar
        │       ├── jackson-mapper-asl-1.8.8.jar
        │       ├── log4j-1.2.16.jar
        │       ├── oro-2.0.8.jar
        │       └── xmlenc-0.52.jar
        ├── maven-archiver
        │   └── pom.properties
        └── oozie-hadoop-1.1.1.oozie-3.3.2.jar
    
    5 directories, 23 files
    MacBookPro:hadooplibs gsun$ 

     
  • Create a directory named libext/ in $OOZIE_HOME, copy all of jars inside hadoop-1 into $OOZIE_HOME/libext/
  • Building with ExtJS library for Oozie Web Console. Copy the extjs-2.2.zip into $OOZIE_HOME/webapp/src/main/webapp, then unzip it.
  • Run oozie-setup.sh to create an oozie.war file.   

    > cd $OOZIE_HOME 
    > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie-setup.sh  -extjs $OOZIE_HOME/webapp/src/main/webapp/ext-2.2.zip

     If the build success, go to the next step, or refer to apache Oozie official docs for oozie-setup.sh usage.
  • Copy the newly minted oozie.war file to your Tomcat deployment directory.

    > cd $OOZIE_HOME
    > vi ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-site.xml

     Change the property below to true as shown.

    <property>
            <name>oozie.service.JPAService.create.db.schema</name>
            <value>true</value>
            <description>
                Creates Oozie DB.
                If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
                If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.
            </description>
        </property>

     
  • Run below command:

    > cd $OOZIE_HOME 
    > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/ooziedb.sh create -sqlfile oozie.sql -run

    if the command is successfully run, the SQL has been created successfully. Start the oozie server.

> cd $OOZIE_HOME
./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozied.sh run

• •  From quick start in Oozie official site,  we know that all Oozie server scripts (=oozie-setup.sh=, oozied.sh , oozie-start.sh , oozie-run.sh and oozie-stop.sh ) run only under the Unix user that owns the Oozie installation directory, if necessary use sudo -u OOZIE_USER when invoking the scripts.As of Oozie 3.3.2, use of oozie-start.sh , oozie-run.sh , and oozie-stop.sh has been deprecated and will print a warning. The oozied.sh script should be used instead; passing it start , run , or stop as an argument will perform the behaviors of oozie-start.sh , oozie-run.sh , and oozie-stop.sh respectively.Check the Oozie log file  ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/oozie.log to ensure Oozie started properly.

 

• • Using the Oozie command line tool check the status of Oozie:

 

• • $ ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie admin -oozie http://[ip address of namenode]:11000/oozie -statusIn my case, the Oozie server can not start due to below exception:

java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils

•  You have to copy hadoop-core.jar and commons-configuration.jar under $HADOOP_HOME/lib into $OOZIE_HOME/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib . You should see the oozie server up and running. Enjoy oozie!

Another problem I met during development is that oozie thrown below exception after job submitted:

Exception in thread "pool-2-thread-25" java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException
         at org.apache.oozie.service.KerberosHadoopAccessorService$1.run(KerberosHadoopAccessorService.java:129)
         at org.apache.oozie.service.KerberosHadoopAccessorService$1.run(KerberosHadoopAccessorService.java:127)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:396)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
         at org.apache.oozie.service.KerberosHadoopAccessorService.createJobClient(KerberosHadoopAccessorService.java:127)
         at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:789)
         at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:534)
         at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:746)
         at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:193)
         at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:53)
         at org.apache.oozie.command.XCommand.call(XCommand.java:257)
         at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:163)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException
         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
         ... 16 more

To solve this problem, you have to copy  jackson-core-asl.jar 和 jackson-mapper-asl.jar from $HADOOP_HOME/lib to $OOZIE_HOME/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib.