搭建hadoop cdh4.5版笔记

1.hadoop cdh各子项目列表，可以手动下载安装配置，也可以yum安装。http://archive.cloudera.com/cdh4/cdh/4/
2.下载各版本压缩包，解压后放置到相应目录，配置环境变量信息修改目录/etc/profile,在文件最后添加各项目安装路径
export JAVA_HOME=/usr/java/jdk1.7.0_09/
export JRE_HOME=${JAVA_HOME}/jre
export HADOOP_HOME=/opt/cdh4.5/hadoop-2.0.0-cdh4.5.0
export HBASE_HOME=/opt/cdh4.5/hbase-0.94.6-cdh4.5.0
export ZOOKEEPER_HOME=/opt/cdh4.5/zookeeper-3.4.5-cdh4.5.0
export PIG_HOME=/opt/cdh4.5/pig-0.11.0-cdh4.5.0
export HIVE_HOME=/opt/cdh4.5/hive-0.10.0-cdh4.5.0
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HBASE_HOME}/bin:${ZOOKEEPER_HOME}/bin:${PIG_HOME}/bin:${HIVE_HOME}/bin:$PATH

3.Hadoop4.5较之前在架构上有了很大的改变。目录结构以前的相比也有了很大的不同。以下是CDH4.5版本的目录结构：


其中：


（1）bin目录下是内层的调用脚本，在进行格式化namenode和格式化ZK等操作的时候，会用到这里的hadoop脚本或者hdfs脚本，要注意更改其中的关于jdk的目录设置和其他必要的设置;


（2）sbin目录下是外层的调用脚本，包括启动整个集群的start-dfs.sh脚本等都在此目录下;


（3）lib目录下是native的so文件;


（4）libexec下是配置程序文件;


（5）etc下是配置文件，和老版本的conf目录对应，比如core-site.xml、slaves和hdfs-site.xml文件等都在此目录下。

（6）share目录下是存放的所有的jar包。

[root@master hadoop-2.0.0-cdh4.5.0]# ls -l
total 48
drwxr-xr-x  2 root root 4096 Dec 30 17:01 bin
drwxr-xr-x  2 root root 4096 Dec 30 17:01 bin-mapreduce1
drwxr-xr-x  3 root root 4096 Dec 30 17:02 cloudera
drwxr-xr-x  6 root root 4096 Dec 30 17:01 etc
drwxr-xr-x  4 root root 4096 Dec 30 17:01 examples
drwxr-xr-x  3 root root 4096 Dec 30 17:01 examples-mapreduce1
drwxr-xr-x  2 root root 4096 Dec 30 17:01 include
drwxr-xr-x  3 root root 4096 Dec 30 17:01 lib
drwxr-xr-x  2 root root 4096 Dec 30 17:01 libexec
drwxr-xr-x  3 root root 4096 Dec 30 17:01 sbin
drwxr-xr-x  4 root root 4096 Dec 30 17:01 share
drwxr-xr-x 16 root root 4096 Dec 30 17:01 src

<span style="font-family:Arial, Helvetica, sans-serif;">熟悉了4.5目录结构后，可以去配置相应的配置文件，在/</span><span style="font-family:Arial, Helvetica, sans-serif;">opt/cdh4.5/hadoop-2.0.0-cdh4.5.0/etc/hadoop目录下</span>

配置core-site.xml

<span style="font-family:Arial, Helvetica, sans-serif;"><property>  </span>

<span style="font-family:Arial, Helvetica, sans-serif;"><name>fs.default.name</name>  </span>

<pre code_snippet_id="136235" snippet_file_name="blog_20131231_7_1224390" name="code" class="html">    <value>hdfs://master:9000</value>
    <final>true</final>
</property> 
 
<property>
    <name>hadoop.tmp.dir</name>
    <value>/hadoop/tmp</value>
</property>

配置hdfs-site.xml

<property>
    <name>dfs.namenode.name.dir</name>
    <value>/hadoop/name</value>
    <final>true</final>
</property>

<property>
    <name>dfs.datanode.data.dir</name>
    <value>/hadoop/data</value>
    <final>true</final>
</property>

<property>
   <name>dfs.replication</name>
   <value>3</value>
   <final>true</final>
</property>

<property>
    <name>dfs.permissions</name>
    <value>false</value>
    <final>true</final>
</property>

配置mapred-site.xml

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>

<property>
    <name>mapreduce.job.tracker</name>
    <value>master:9001</value>
    <final>true</final>
</property>

<property>
    <name>mapred.system.dir</name>
    <value>file:/hadoop/mapred_system</value>
    <final>true</final>
</property>

<property>
    <name>mapred.local.dir</name>
    <value>file:/hadoop/mapred_local</value>
    <final>true</final>
</property>

</pre><pre code_snippet_id="136235" snippet_file_name="blog_20131231_9_7604102" name="code" class="html">

配置yarn-site.xml

查看<a target=_blank target="_blank" href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml">http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml</a>了解yarn默认端口配置。
<pre code_snippet_id="136235" snippet_file_name="blog_20131231_7_1224390" name="code" class="html"><property>
    <name>yarn.resourcemanager.address</name>
    <value>master:<span style="font-family:Simsun;font-size:14px;">8032</span></value>
</property>

<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:<span style="font-family:Simsun;font-size:14px;">8030</span></value>
</property>

<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:<span style="font-family:Simsun;font-size:14px;">8031</span></value>
</property>

<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:<span style="font-family:Simsun;font-size:14px;">8033</span></value>
</property>

<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:8088</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce.shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

[root@master ~]# tree /hadoop/
/hadoop/
|-- data
|-- mapredlocal
|-- mapredsystem
|-- name
`-- tmp

4.把master配置好的目录以及文件拷贝到slave1，slave2