1.hadoop cdh各子项目列表,可以手动下载安装配置,也可以yum安装。http://archive.cloudera.com/cdh4/cdh/4/
2.下载各版本压缩包,解压后放置到相应目录,配置环境变量信息修改目录/etc/profile,在文件最后添加各项目安装路径
export JAVA_HOME=/usr/java/jdk1.7.0_09/
export JRE_HOME=${JAVA_HOME}/jre
export HADOOP_HOME=/opt/cdh4.5/hadoop-2.0.0-cdh4.5.0
export HBASE_HOME=/opt/cdh4.5/hbase-0.94.6-cdh4.5.0
export ZOOKEEPER_HOME=/opt/cdh4.5/zookeeper-3.4.5-cdh4.5.0
export PIG_HOME=/opt/cdh4.5/pig-0.11.0-cdh4.5.0
export HIVE_HOME=/opt/cdh4.5/hive-0.10.0-cdh4.5.0
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HBASE_HOME}/bin:${ZOOKEEPER_HOME}/bin:${PIG_HOME}/bin:${HIVE_HOME}/bin:$PATH
3.Hadoop4.5较之前在架构上有了很大的改变。目录结构以前的相比也有了很大的不同。以下是CDH4.5版本的目录结构:
其中:
(1)bin目录下是内层的调用脚本,在进行格式化namenode和格式化ZK等操作的时候,会用到这里的hadoop脚本或者hdfs脚本,要注意更改其中的关于jdk的目录设置和其他必要的设置;
(2)sbin目录下是外层的调用脚本,包括启动整个集群的start-dfs.sh脚本等都在此目录下;
(3)lib目录下是native的so文件;
(4)libexec下是配置程序文件;
(5)etc下是配置文件,和老版本的conf目录对应,比如core-site.xml、slaves和hdfs-site.xml文件等都在此目录下。
(6)share目录下是存放的所有的jar包。
[root@master hadoop-2.0.0-cdh4.5.0]# ls -l
total 48
drwxr-xr-x 2 root root 4096 Dec 30 17:01 bin
drwxr-xr-x 2 root root 4096 Dec 30 17:01 bin-mapreduce1
drwxr-xr-x 3 root root 4096 Dec 30 17:02 cloudera
drwxr-xr-x 6 root root 4096 Dec 30 17:01 etc
drwxr-xr-x 4 root root 4096 Dec 30 17:01 examples
drwxr-xr-x 3 root root 4096 Dec 30 17:01 examples-mapreduce1
drwxr-xr-x 2 root root 4096 Dec 30 17:01 include
drwxr-xr-x 3 root root 4096 Dec 30 17:01 lib
drwxr-xr-x 2 root root 4096 Dec 30 17:01 libexec
drwxr-xr-x 3 root root 4096 Dec 30 17:01 sbin
drwxr-xr-x 4 root root 4096 Dec 30 17:01 share
drwxr-xr-x 16 root root 4096 Dec 30 17:01 src
<span style="font-family:Arial, Helvetica, sans-serif;">熟悉了4.5目录结构后,可以去配置相应的配置文件,在/</span><span style="font-family:Arial, Helvetica, sans-serif;">opt/cdh4.5/hadoop-2.0.0-cdh4.5.0/etc/hadoop目录下</span>
配置core-site.xml
<span style="font-family:Arial, Helvetica, sans-serif;"><property> </span>
<span style="font-family:Arial, Helvetica, sans-serif;"><name>fs.default.name</name> </span>
<pre code_snippet_id="136235" snippet_file_name="blog_20131231_7_1224390" name="code" class="html"> <value>hdfs://master:9000</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
配置hdfs-site.xml
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/name</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
<final>true</final>
</property>
配置mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.job.tracker</name>
<value>master:9001</value>
<final>true</final>
</property>
<property>
<name>mapred.system.dir</name>
<value>file:/hadoop/mapred_system</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>file:/hadoop/mapred_local</value>
<final>true</final>
</property>
</pre><pre code_snippet_id="136235" snippet_file_name="blog_20131231_9_7604102" name="code" class="html">
配置yarn-site.xml
查看<a target=_blank target="_blank" href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml">http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml</a>了解yarn默认端口配置。
<pre code_snippet_id="136235" snippet_file_name="blog_20131231_7_1224390" name="code" class="html"><property>
<name>yarn.resourcemanager.address</name>
<value>master:<span style="font-family:Simsun;font-size:14px;">8032</span></value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:<span style="font-family:Simsun;font-size:14px;">8030</span></value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:<span style="font-family:Simsun;font-size:14px;">8031</span></value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:<span style="font-family:Simsun;font-size:14px;">8033</span></value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
[root@master ~]# tree /hadoop/
/hadoop/
|-- data
|-- mapredlocal
|-- mapredsystem
|-- name
`-- tmp
4.把master配置好的目录以及文件拷贝到slave1,slave2