hadoop-2.6.0集群环境搭建

一、主机规划

            1、准备4台Ubuntu 14.04 64-bit 虚拟机，一台充当resourcemanager和namenode，另外三台充当nodemanager和datanode。由于需要实现主机间ssh无密码访问，主机IP采用静态配置。配置如下：

namenode  ip:192.168.1.110

datanode1 ip:192.168.1.111

datanode2 ip:192.168.1.112

datanode3 ip:192.168.1.113

分别修改每一台的主机名和hosts文件

$sudo vim /etc/hostname

$sudo vim /etc/hosts

2、新建用户组和用户

$sudo groupadd cluster

$sudo useradd -m -s /bin/bash -g cluster -G sudo hadoop

$sudo passwd hadoop

sudo usermod -a -G adm hadoop

sudo usermod -a -G sudo hadoop

注销当前用户以hadoop用户登陆，主要是方便之后使用gedit编辑器修改配置文件。

3、安装ssh并配置无密码访问，依次执行下面的命令：

$ sudo apt-get install ssh

$ sudo apt-get install rsync

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

4、配置namenode无密码访问datanode1

在datanode1上切换到/home/hadoop/.ssh执行

$ scp hadoop@namenode:/home/hadoop/.ssh/id_dsa.pub ./namenode_dsa.pub

$ cat namenode_dsa.pub >>authorized_keys

在namenode上执行：

$ ssh hadoop@datanode1(第一次需要输入密码，之后便可无密码访问）

同上分别配置datanode2 和datanode3

二、安装jdk和hadoop-2.6.0

1、安装jdk

到Oracle官网下载jdk-8u25-Linux-x64.tar.gz将其拷贝到/usr目录，执行：$ sudo tar -zxf /usr/jdk-8u25-linux-x64.tar.gz

2、安装hadoop-2.6.0

到http://hadoop.apache.org/下载hadoop-2.6.0.tar.gz拷贝到/home/hadoop目录，执行：$ tar -zxf /home/hdoop/hadoop-2.6.0.tar.gz

3、配置环境变量

执行：$ sudo vim /etc/profile 在文件末尾添加如下内容：

export JAVA_HOME=/usr/jdk1.8.0_25
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_PREFIX=/home/hadoop/hadoop-2.6.0
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.6.0/etc/hadoop
export HADOOP_YARN_HOME=/home/hadoop/hadoop-2.6.0

三、配置hadoop

1、配置/home/hadoop/hadoop-2.6.0/etc/hadoop/core-site.xml

[html]  view plain  copy

1. <?xml version="1.0" encoding="UTF-8"?>  
2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
3. <configuration>  
4.     <property>  
5.         <name>fs.defaultFS</name>  
6.         <value>hdfs://namenode:9000</value>  
7.     </property>  
8.     <property>  
9.         <name>io.file.buffer.size</name>  
10.         <value>131072</value>  
11.     <description>Size of read/write buffer used in SequenceFiles.</description>  
12.     </property>  
13. </configuration>  

2、配置/home/hadoop/hadoop-2.6.0/etc/hadoop/hdfs-site.xml

[html]  view plain  copy

1. <?xml version="1.0" encoding="UTF-8"?>  
2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
3. <configuration>  
4.     <property>  
5.         <name>dfs.replication</name>  
6.         <value>2</value>  
7.     </property>  
8.     <property>  
9.         <name>dfs.namenode.name.dir</name>  
10.         <value>/home/hadoop/hdfs/name</value>  
11.     <description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.</description>  
12.     </property>  
13.     <property>  
14.         <name>dfs.datanode.data.dir</name>  
15.         <value>/home/hadoop/hdfs/data</value>  
16.     </property>  
17.     <property>  
18.         <name>dfs.blocksize</name>  
19.         <value>268435456</value>  
20.     </property>  
21.     <property>  
22.         <name>dfs.namenode.handler.count</name>  
23.         <value>100</value>  
24.     </property>  
25. </configuration>  

3、配置/home/hadoop/hadoop-2.6.0/etc/hadoop/yarn-site.xml

[html]  view plain  copy

1. <?xml version="1.0"?>  
2. <configuration>  
3.     <property>  
4.         <name>yarn.acl.enable</name>  
5.         <value>true</value>  
6.     </property>  
7.     <property>  
8.         <name>yarn.admin.acl</name>  
9.         <value>*</value>  
10.         <description>ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which means anyone. Special value of just space means no one has access.</description>  
11.     </property>  
12.     <property>  
13.         <name>yarn.log-aggregation-enable</name>  
14.         <value>false</value>  
15.         <description>Configuration to enable or disable log aggregation</description>  
16.     </property>  
17.     <property>  
18.         <name>yarn.resourcemanager.address</name>  
19.         <value>192.168.1.110:8031</value>  
20.     </property>  
21.     <property>  
22.         <name>yarn.resourcemanager.scheduler.address</name>  
23.         <value>192.168.1.110:8032</value>  
24.     </property>  
25.     <property>  
26.         <name>yarn.resourcemanager.resource-tracker.address</name>  
27.         <value>192.168.1.110:8033</value>  
28.     </property>  
29.     <property>  
30.         <name>yarn.resourcemanager.admin.address</name>  
31.         <value>192.168.1.110:8034</value>  
32.     </property>  
33.     <property>  
34.         <name>yarn.resourcemanager.webapp.address</name>  
35.         <value>192.168.1.110:8088</value>  
36.     </property>  
37.     <property>  
38.         <name>yarn.resourcemanager.hostname</name>  
39.         <value>192.168.1.110</value>  
40.     </property>  
41.     <property>  
42.         <name>yarn.scheduler.minimum-allocation-mb</name>  
43.         <value>256</value>  
44.     </property>  
45.     <property>  
46.         <name>yarn.scheduler.maximum-allocation-mb</name>  
47.         <value>6144</value>  
48.     </property>  
49.     <property>  
50.         <name>yarn.nodemanager.resource.memory-mb</name>  
51.         <value>256</value>  
52.     </property>  
53.     <property>  
54.         <name>yarn.nodemanager.vmem-pmem-ratio</name>  
55.         <value>20</value>  
56.     </property>  
57.     <property>  
58.         <name>yarn.nodemanager.local-dirs</name>  
59.         <value>/home/hadoop/comma</value>  
60.     </property>  
61.     <property>  
62.         <name>yarn.nodemanager.log-dirs</name>  
63.         <value>/home/hadoop/log</value>  
64.     </property>  
65.     <property>  
66.         <name>yarn.nodemanager.log.retain-seconds</name>  
67.         <value>10800</value>  
68.     </property>  
69.     <property>  
70.         <name>yarn.nodemanager.remote-app-log-dir</name>  
71.         <value>/logs</value>  
72.     </property>  
73.     <property>  
74.         <name>yarn.nodemanager.remote-app-log-dir-suffix</name>  
75.         <value>logs</value>  
76.     </property>  
77.     <property>  
78.         <name>yarn.nodemanager.aux-services</name>  
79.         <value>mapreduce_shuffle</value>  
80.     </property>  
81.     <property>  
82.         <name>yarn.log-aggregation.retain-seconds</name>  
83.         <value>-1</value>  
84.     </property>  
85.     <property>  
86.         <name>yarn.log-aggregation.retain-check-interval-seconds</name>  
87.         <value>-1</value>  
88.     </property>  
89. </configuration>  

4、配置/home/hadoop/hadoop-2.6.0/etc/hadoop/mapred-site.xml

[html]  view plain  copy

1. <?xml version="1.0"?>  
2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
3. <configuration>  
4.     <property>  
5.         <name>mapreduce.framework.name</name>  
6.         <value>yarn</value>  
7.     </property>  
8.     <property>  
9.         <name>mapreduce.map.memory.mb</name>  
10.         <value>1536</value>  
11.     </property>  
12.     <property>  
13.         <name>mapreduce.map.java.opts</name>  
14.         <value>-Xmx1024M</value>  
15.     </property>  
16.     <property>  
17.         <name>mapreduce.reduce.memory.mb</name>  
18.         <value>3072</value>  
19.     </property>  
20.     <property>  
21.         <name>mapreduce.reduce.java.opts</name>  
22.         <value>-Xmx2560M</value>  
23.     </property>  
24.     <property>  
25.         <name>mapreduce.task.io.sort.mb</name>  
26.         <value>512</value>  
27.     </property>  
28.     <property>  
29.         <name>mapreduce.task.io.sort.factor</name>  
30.         <value>100</value>  
31.     </property>  
32.     <property>  
33.         <name>mapreduce.reduce.shuffle.parallelcopies</name>  
34.         <value>50</value>  
35.     </property>  
36.     <property>  
37.         <name>mapreduce.jobhistory.address</name>  
38.         <value>192.168.1.110:10020</value>  
39.     </property>  
40.     <property>  
41.         <name>mapreduce.jobhistory.webapp.address</name>  
42.         <value>192.168.1.110:19888</value>  
43.     </property>  
44.     <property>  
45.         <name>mapreduce.jobhistory.intermediate-done-dir</name>  
46.         <value>/mr-history/tmp</value>  
47.     </property>  
48.     <property>  
49.         <name>mapreduce.jobhistory.done-dir</name>  
50.         <value>/mr-history/done</value>  
51.     </property>  
52. </configuration>  

5、配置/home/hadoop/hadoop-2.6.0/etc/hadoop/slaves 把datanode的ip或主机名一行一个写入文件中

datanode1 或者192.168.1.111

datanode2 或者192.168.1.112

datanode3 或者192.168.1.113

6、启动hadoop

在namenode主机上启动如下进程：

格式化hdfs文件系统

$ $HADOOP_PREFIX/bin/hdfs namenode -format

启动namenode

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode

启动resourcemanager

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

启动MapReduce JobHistory Server

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR

使用jps命令查看运行情况：

分别在每台datanode上启动：

启动datanode

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

启动nodemanager

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager

7、停止服务

要停止相应的服务只需把上面的命令中的start改为stop即可。

可通过浏览器访问如下地址查看hadoop的运行状态：

NameNode http://namenode:50070

ResourceManager http://namenode:8088

MapReduce JobHistory Server http://namenode:19888

四、运行测试

1、在hdfs文件系统上新建输入、输出目录

$ $HADOOP_PREFIX/bin/hdfs -mkdir /input

$ $HADOOP_PREFIX/bin/hdfs -mkdir /output

此次用hadoop自带的wordcount来做测试，在当前目录下新建测试文件test.txt

$ touch test.txt 并向文件中写入一定量的单词文本

$ $HADOOP_PREFIX/bin/hdfs -copyFromLocal test.txt /input

运行程序

$ $HADOOP_PREFIX/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input/ /output/result

查看结果

$ $HADOOP_PREFIX/bin/hdfs -cat /output/result/*

Web 页面地址

hadoop集群启动成功后，可以通过以下web页面地址查看几个组件的运行情况：

Daemon	Web Interface	Notes
NameNode	http://host:port/	Default HTTP port is 50070.
ResourceManager	http://host:port/	Default HTTP port is 8088.
MapReduce JobHistory Server	http://host:port/	Default HTTP port is 19888.