一:简介
二:搭建CDH版本的hadoop集群
- 所需安装包准备以及环境
3台机器:
192.168.191.111 hadoop111
192.168.191.112 hadoop112
192.168.191.113 hadoop113 - 将安装包上传到Linux
- 首先创建cdh文件夹,并且将cdh版本的hadoop解压进去
- 修改slaves配置文件(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下),配置各个节点名称
- 修改yarn-env.sh(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 修改mapred-env.sh(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 修改hadoop-env.sh(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 修改core-site.xml(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 修改hdfs-site.xml(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 修改 mapred-site.xml(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
下面的历史服务器是为Oozie准备的,必须要配置。
- 修改 yarn-site.xml(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 整合Oozie需要配置的内容,修改core-site.xml(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
注意:kgf指的是安装Oozie时用的用户
- 整合Oozie需要配置的内容,修改yarn-site.xml(/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop目录下)
- 最后将配置文件分发到其它机器上
- 第一次启动需要格式化namenode,最好在/etc/profile中配置hadoop地址
- 启动Hadoop集群
a:启动dfs,在hadoop111机器
b:在hadoop112上启动yarn
c:启动历史服务器,在hadoop111机器上
d:效果
三:安装Oozie
- 解压Oozie
- 在oozie根目录下解压oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz
进入/usr/local/module/oozie-4.0.0-cdh5.3.6/oozie-4.0.0-cdh5.3.6目录下,将hadooplibs移动到
/usr/local/module/oozie-4.0.0-cdh5.3.6目录下:
- 在Oozie目录下创建libext目录
- 将hadooplibs里面的jar包,拷贝到libext目录下
- 拷贝Mysql驱动包到libext目录下
- 将ext-2.2.zip拷贝到libext/目录下(ext是一个js框架,用于展示oozie前端页面)
- 修改Oozie配置文件,/usr/local/module/oozie-4.0.0-cdh5.3.6/conf目录下oozie-site.xml文件
<!--配置mysql驱动--> <property> <name>oozie.service.JPAService.jdbc.driver</name> <value>com.mysql.jdbc.Driver</value> <description> JDBC driver class. </description> </property> <!--配置mysql url地址--> <property> <name>oozie.service.JPAService.jdbc.url</name> <value>jdbc:mysql://hadoop111:3306/oozie</value> <description> JDBC URL. </description> </property> <!--配置mysql用户名--> <property> <name>oozie.service.JPAService.jdbc.username</name> <value>root</value> <description> DB user name. </description> </property> <!--配置mysql密码--> <property> <name>oozie.service.JPAService.jdbc.password</name> <value>897570</value> <description> DB user password. IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value, if empty Configuration assumes it is NULL. </description> </property> <!--让Oozie引用Hadoop的配置文件--> <property> <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> <value>*=/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop</value> <description> Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is used when there is no exact match for an authority. The HADOOP_CONF_DIR contains the relevant Hadoop *-site.xml files. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. </description> </property>
- 进入Mysql并创建oozie数据库
- 初始化Oozie
⑴上传Oozie目录下的yarn.tar.gz文件到HDFS
a:命令
bin/oozie-setup.sh sharelib create -fs hdfs://hadoop111:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz
效果:
⑵创建oozie.sql文件,在/usr/local/module/oozie-4.0.0-cdh5.3.6/目录下
命令:bin/ooziedb.sh create -sqlfile oozie.sql -run
- 打包项目,生成war包,在/usr/local/module/oozie-4.0.0-cdh5.3.6/目录下
命令:bin/oozie-setup.sh prepare-war
- 启动Oozie,在/usr/local/module/oozie-4.0.0-cdh5.3.6/目录下
命令:bin/oozied.sh start,关闭命令:bin/oozied.sh stop
- Oozie页面效果