【Oozie笔记】2. Oozie的部署

简介

这一篇文章主要写如何部署Oozie，看了一下发现Oozie的包比hadoop的包都大，确实是一个体量很大的定时调度任务框架。

前期准备

包准备：

1. ext-2.2.zip(一个JS的框架，主要用于Oozie前端的支持)
2. hadoop-2.5.0-cdh5.3.6.tar.gz（使用的是CDH版本的Hadoop）
3. oozie-4.0.0-cdh5.3.6.tar.gz （也是CDH版本的Oozie）
4. mysql-connector-java-5.1.27-bin.jar

这些包不提供下载，直接去百度上找就可以了。

环境准备：
Ubuntu 14.04.5 LTS \n \l

安装CDH的Hadoop

1. 在目录/opt/software上传这三个准备好的包
   

2. 在目录/opt/module目录下创建cdh目录，并把hadoop解压在该目录。把Oozie的压缩包解压在/opt/module
tar -zxvf hadoop-2.5.0-cdh5.3.6.tar.gz -C /opt/module/cdh/
tar -zxvf oozie-4.0.0-cdh5.3.6.tar.gz -C /opt/module/
   

3. 修改CDH版本的Hadoop的配置文件
   需要修改的8个文件分是：
   hadoop-env.sh
   修改JAVA_HOME
   export JAVA_HOME=/usr/soft/java/jdk1.8.0_201
   yarn-env.sh
   添加JAVA_HOME
   export JAVA_HOME=/usr/soft/java/jdk1.8.0_201
   mapred-env.sh
   添加JAVA_HOME
   export JAVA_HOME=/usr/soft/java/jdk1.8.0_201
   hdfs-site.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   <!--
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
     You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License. See accompanying LICENSE file.
   -->
   
   <!-- Put site-specific property overrides in this file. -->
   <configuration>
    	<property>
           <name>dfs.replication</name>
           <value>1</value>
           <description>复制的块（即数据节点）的数量</description>
       </property>
       <property>
           <name>dfs.namenode.secondary.http-address</name>
           <value>master:50090</value>
           <description>辅助管理节点HTTP服务器地址和端口</description>
       </property>
   </configuration>
   

   mapred-site.xml

   <?xml version="1.0"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   <!--
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
     You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License. See accompanying LICENSE file.
   -->
   
   <!-- Put site-specific property overrides in this file. -->
   
   <configuration>
          <property>
                  <name>mapreduce.framework.name</name>
                   <value>yarn</value>
                   <description>执行MapReduce作业时运行的框架</description>
           </property>
           <property>
                   <name>mapreduce.jobhistory.address</name>
                   <value>master:10020</value>
                   <description>MapReduce jobhistory服务器的进程间通信地址</description>
           </property>
           <property>
                   <name>mapreduce.jobhistory.webapp.address</name>
                   <value>master:19888</value>
                   <description>MapReduce jobhistory服务器的用户界面地址</description>
           </property>
   </configuration>
   

   core-site.xml

   	<?xml version="1.0" encoding="UTF-8"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   <!--
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
     You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License. See accompanying LICENSE file.
   -->
   
   <!-- Put site-specific property overrides in this file. -->
   
   <configuration>
   
   	<property>
               <name>hadoop.tmp.dir</name>
               <value>file:/opt/module/cdh/hadoop-2.5.0-cdh5.3.6/tmp</value>
               <description>Hadoop重要临时文件存放目录</description>
       </property>
       <property>
               <name>fs.defaultFS</name>
               <value>hdfs://master:8020</value>
               <description>一种方案和权限决定文件系统实现的URI</description>
       </property>
   	<!-- Oozie Server的Hostname -->
   	<property>
   		<name>hadoop.proxyuser.root.hosts</name>
   		<value>*</value>
   		<description>代理：*代表Oozie安装的主机，root代表用户名</description>
   	</property>
   
   	<!-- 允许被Oozie代理的用户组 -->
   	<property>
   		<name>hadoop.proxyuser.root.groups</name>
   	 	<value>*</value>
   	 	<description>用户组</description>
   	</property>
   
   </configuration>
   

   yarn-site.xml

   <?xml version="1.0"?>
   <!--
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
     You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License. See accompanying LICENSE file.
   -->
   <configuration>
   
   <!-- Site specific YARN configuration properties -->
   	<property>
           <name>yarn.resourcemanager.hostname</name>
           <value>master</value>
           <description>ResourceManager的主机名</description>
       </property>
       <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
           <description>NodeManager的辅助服务</description>
       </property>
       <property>
           <name>yarn.log-aggregation-enable</name>
           <value>true</value>
           <description>启用日志聚合，默认值为False，即禁用</description>
       </property>
       <property>
           <name>yarn.log-aggregation.retain-seconds</name>
           <value>86400</value>
           <description>日志保留时间设置为一天</description>
       </property>
       <property>
           <name>yarn.nodemanager.log-dirs</name>
           <value>${yarn.log.dir}/userlogs</value>
           <description>应用程序的本地化的日志目录</description>
       </property>
   </configuration>
   

   slaves

   master
   

   我这里写master 是因为我的hosts写的master
   

4. 格式化namenode

   因为我是伪分布式，只有一台机器所以少了一步分发步骤。格式化namenode之前，添加环境变量，在/etc/profile修改

   export HADOOP_INSTALL=/opt/module/cdh/hadoop-2.5.0-cdh5.3.6
   export PATH=$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin:$PATH
   export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
   

   修改完之后，source一下
   执行 格式化命令 bin/hdfs namenode -format，一定要带着bin。

   看到这个截图内容就是成功了。
   

5. 启动hadoop集群
   一定要加这sbin 执行命令sbin/start-all.sh
   

6. 启动jobhistoryserver
   执行启动命令sbin/mr-jobhistory-daemon.sh start historyserver
   
   没有这个 日志就看不了了
   至此CDH的Hadoop部署完成。打开HDFS的web界面
   

安装Oozie

上一步我们已经解压了Oozie的压缩包，在/opt/module/目录下。

还有一部就是要解压oozie需要用到的hadoop的lib包tar -zxvf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz -C ../

1. 在Oozie目录下创建libext目录
   目录名一定是libext！！！！
   root@master:/opt/module/oozie-4.0.0-cdh5.3.6# mkdir libext

2. 拷贝依赖的jar包
   注意拷贝的是jar包！！！
   cp hadooplibs/hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6/* /opt/module/oozie-4.0.0-cdh5.3.6/libext/
   还需要拷贝的是ext的包和mysql的驱动包，ext的直接就是zip包。不需要解压等其他操作
   cp /opt/software/ext-2.2.zip /opt/module/oozie-4.0.0-cdh5.3.6/libext/
   mysql-connector-java-5.1.27-bin.jar自行百度下载
   

3. 修改Oozie配置文件
   修改/opt/module/hive/conf/oozie-site.xml 配置文件，查找属性替换对应的值

   属性：oozie.service.JPAService.jdbc.driver
   属性值：com.mysql.jdbc.Driver
   解释：JDBC的驱动
   // 这里的master是我的机器名 在 hosts中定义了本机叫master
   属性：oozie.service.JPAService.jdbc.url
   属性值：jdbc:mysql://master:3306/oozie
   解释：oozie所需的数据库地址
   
   属性：oozie.service.JPAService.jdbc.username
   属性值：root
   解释：数据库用户名
   
   属性：oozie.service.JPAService.jdbc.password
   属性值：000000
   解释：数据库密码
   	
   

   还需要让oozie知道hadoop的位置，找到属性：oozie.service.HadoopAccessorService.hadoop.configurations 替换为当前环境的hadoop配置文件的地址(*= 不要删掉)
   

4. 创建oozie的数据库
   mysql -uroot -p123456
mysql> create database oozie;
   

5. 初始化Oozie
   1）上传Oozie目录下的oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz 文件到HDFS
   注意：该文件会自行解压
   执行命令：bin/oozie-setup.sh sharelib create -fs hdfs://master:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz
   运行日志：
   去HDFS的web界面检查一下
   
   2）运行oozie.sql文件，初始化数据库
   bin/ooziedb.sh create -sqlfile oozie.sql -run 有可能遇到连接mysql数据库失败
   
   解决办法：修改mysql安装目录中my.cnf文件
   修改bind-address = 127.0.0.1 为 bind-address = 0.0.0.0
   然后重启mysql数据库
   重新执行bin/ooziedb.sh create -sqlfile oozie.sql -run
   
   数据库查看表：
   
   3）打包项目，生成war包
   oozie需要打成war包，执行bin/oozie-setup.sh prepare-war
   
   可以看到已经准备好了，就差我们启动oozie了。

4）启动Oozie
执行 bin/oozied.sh start ,运行日志：
bin/oozied.sh stop是停止
看进程有Bootstrap就可以了：

至此 我们就搭建完成了，查看oozie的页面，地址： http://192.168.2.131:11000/oozie/