因为需要分析大量的访问日志，现有的分析方式（单机）不能满足需求，因此需要部署hadoop来解决这个问题。在我做分布式文件系统之前，曾经部署测试过hadoop，使用的版本为hadoop-0.19.0，记得当时配置起来很快就成功了。这次测试，我使用的是最新版本hadoop-0.20.1,在部署过程中，花了不少时间才做成功（2天）。为备忘，特记录之。
hadoop-0.20.1跟旧的版相比，有些文件发生了变化，主要的变化是conf目录，我在这里给出一个简单的对比表：
| Hadoop-0.19.0 | Hadoop-0.20.1 |
| ll hadoop-0.19.0/conf/ | ll hadoop-0.20.1/conf/ |
| total 92-rw-rw-r-- 1 root root 2065 Nov 14 2008 capacity-scheduler.xml-rw-rw-r-- 1 root root 535 Nov 14 2008 configuration.xsl-rw-rw-r-- 1 root root 49456 Nov 14 2008 hadoop-default.xml-rw-rw-r-- 1 root root 2237 Nov 14 2008 hadoop-env.sh-rw-rw-r-- 1 root root 1245 Nov 14 2008 hadoop-metrics.properties-rw-rw-r-- 1 root root 178 Nov 14 2008 hadoop-site.xml-rw-rw-r-- 1 root root 2815 Nov 14 2008 log4j.properties-rw-rw-r-- 1 root root 10 Nov 14 2008 masters-rw-rw-r-- 1 root root 10 Nov 14 2008 slaves-rw-rw-r-- 1 root root 401 Nov 14 2008 sslinfo.xml.example | total 56-rw-rw-r-- 1 root root 3936 Sep 2 04:44 capacity-scheduler.xml-rw-rw-r-- 1 root root 535 Sep 2 04:44 configuration.xsl-rw-rw-r-- 1 root root 178 Sep 2 04:44 core-site.xml-rw-rw-r-- 1 root root 2237 Sep 2 04:44 hadoop-env.sh-rw-rw-r-- 1 root root 1245 Sep 2 04:44 hadoop-metrics.properties-rw-rw-r-- 1 root root 4190 Sep 2 04:44 hadoop-policy.xml-rw-rw-r-- 1 root root 178 Sep 2 04:44 hdfs-site.xml-rw-rw-r-- 1 root root 2815 Sep 2 04:44 log4j.properties-rw-rw-r-- 1 root root 178 Sep 2 04:44 mapred-site.xml-rw-rw-r-- 1 root root 10 Sep 2 04:44 masters
-rw-rw-r-- 1 root root 10 Sep 2 04:44 slaves
-rw-rw-r-- 1 root root 1243 Sep 2 04:44 ssl-client.xml.example
-rw-rw-r-- 1 root root 1195 Sep 2 04:44 ssl-server.xml.example
|
网上现有的hadoop文章，需要修改的文件是hadoop-site.xml,可是hadoop-0.20.1版本确没有这个文件，取而代之的是core-site.xml。
一、部署hadoop
hadoop部署分两步：名称节点（namenode）无密码访问各数据节点(datanode)和配置hadoop.我的实验环境为一个名称节点，两个数据节点.不幸的是其中的一个数据节点坏了。因此只能用一个数据节点来测试。在这个测试中，namenode的ip:192.168.199.135，datanode的ip:192.168.199.136.
(一)、名称节点（namenode）无密码访问各数据节点(datanode)
1、名称节点和数据节点各自创建用户hadoop,使用相同的密码。
2、以hadoop用户名登陆名称节点执行
ssh-keygen -t rsa 然后一路回车，完毕后生成文件.ssh/id_rsa.pub,把这个文件复制到当前位置，命名为authorized_keys；然后执行命令 ssh 127.0.0.1,如果不需要密码则直接登陆进去的话，就达到要求；否则需检查authorized_keys的权限，看是否为644（-rw-r--r--）。接下来，同样也hadoop用户登陆数据节点服务器，创建.ssh目录，并给与600权限（chmod 600 .ssh）; 再把名称服务器上的authorized_keys复制到目录./ssh,注意权限和目录结构跟名称节点保持一致，然后再从名称节点用ssh登陆数据节点，如果不需要密码能登陆成功，则ssh的配置结束。
(二) hadoop配置
1、下载jdk并设置。我的 JAVA_HOME =/usr/local/jdk1.6.0_06
2、下载hadoop，解包，复制到/usr/local/hadoop.即 HADOOP_HOME=/usr/local/hadoop
3、数据存放目录分离，即数据的实际存放不在 HADOOP_HOME（网上的n多文章是在hadoop安装目录）.我用2个1TB的硬盘单独来存放实际的数据块，其名称为/disk2,/disk3.格式化这两个硬盘并创建文件系统，然后挂接到这2个目录。相关步骤省略。
4、设置目录、文件的属主。chown -R hadoop:hadoop /disk2 /disk3 /usr/local/hadoop
5、在文件/usr/local/hadoop/conf/core-site.xml 的<configuration>与</configuration>中间插入如下内容（内容从网上找的）：
|
<property>
<name>fs.default.name</name>
<value>
hdfs://hadoopm:9000</value>
<description>The name of the default file system. Either the literal string "local" or a host:port for DFS.</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoopm:9001</value>
<description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and
reduce task.</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/disk2/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>/disk2/hadoop/filesystem/name</value>
<description>Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/disk2/hadoop/filesystem/data</value>
<description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are i
gnored.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication. The actual number of replications can be specified when the file is created. The default isused if replication is not specified in create time.</description>
</property>
|
根据字面意思，不难理解每个文本块的作用，因此这里不再啰嗦。有一点需要提及，就是“<value>
hdfs://hadoop:9000</value>”，我起初用的是ip,运行程序时可能会出错。
| STARTUP_MSG: Starting DataNodeSTARTUP_MSG: host = hadoop-s2/127.0.0.1STARTUP_MSG: args = []STARTUP_MSG: version = 0.20.1STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009************************************************************/2009-10-15 10:16:01,592 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-m/192.168.199.135:9000. Already tried 0 time(s).2009-10-15 10:16:02,594 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-m/192.168.199.135:9000. Already tried 1 time(s).………………………………………(省略)……………… |
6、修改文件/usr/local/hadoop/conf/slaves及/usr/local/hadoop/conf/masters,把数据节点的主机名加到slaves、名称节点主机名加到masters。可以加多个，每行一个。注意主机名需要在每个服务器的/etc/hosts映射好。
7、修改文件/usr/local/hadoop/conf/hadoop-env.sh,加入 export JAVA_HOME=/usr/local/jdk1.6.0_06一行即可。
8、在每个数据节点重复1~7这些步骤。
二、初始化和启动hadoop集群
（一）hadoop的大部分操作都是在命名节点进行的。以hadoop用户登陆系统然后执行 hadoop namenode -format ,一般都能顺利完成这个过程。执行完初始化后，各数据节点并没有进行任何操作（最重要的事情就是生成一堆目录）。
（二）启动hadoop服务。命名服务器以hadoop用户执行 start-all.sh . 查看进程，如果正常，应该看见2-3个java进程。启动正常的话，数据节点开始生成相关的目录。其对比输出如下：
| 名称节点未执行start-all.sh数据节点相关目录的情况 | 名称节点执行start-all.sh数据节点相关目录的情况 |
| [root@hadoop-s2 conf]# ll /disk2total 16drwx------ 2 hadoop hadoop 16384 Oct 13 19:53 lost+found | [root@hadoop-s2 conf]# ll /disk2total 20drwxrwxr-x 3 hadoop hadoop 4096 Oct 16 09:42 hadoopdrwx------ 2 hadoop hadoop 16384 Oct 13 19:53 lost+found |
三、测试
1、查看hadoop状态，使用命令 $hadoop dfs -report.下面是我的服务器的输出：
| [hadoop@hadoop-m ~]$ hadoop dfsadmin -reportConfigured Capacity: 984509087744 (916.9 GB)Present Capacity: 933221065008 (869.13 GB)DFS Remaining: 932155265024 (868.14 GB)DFS Used: 1065799984 (1016.43 MB)DFS Used%: 0.11%
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 192.168.193.137:50010
Decommission Status : Normal
Configured Capacity: 984509087744 (916.9 GB)
DFS Used: 1065799984 (1016.43 MB)
Non DFS Used: 51288022736 (47.77 GB)
DFS Remaining: 932155265024(868.14 GB)
DFS Used%: 0.11%
DFS Remaining%: 94.68%
Last contact: Thu Oct 15 15:56:07 CST 2009
|
特别要注意有没有“
Datanodes available: 1 (1 total, 0 dead)”出现，如果没有数据节点被连接，请登录数据节点，查看hadoop的日志文件，查找其失败的原因。
2、创建目录。$hadoop dfs -mkdir sery ,然后再传几个大文件 $hadoop dfs -put 7.* sery . 传了900M 左右的数据（2个iso文件），很快传完。执行 $hadoop dfs -ls sery , 其输出为：
| [hadoop@hadoop-m ~]$ hadoop dfs -ls seryFound 2 items-rw-r--r-- 1 hadoop supergroup 523335680 2009-10-15 15:52 /7.0-RELEASE-amd64-disc1.iso-rw-r--r-- 1 hadoop supergroup 534177792 2009-10-15 15:52 /7.0-RELEASE-i386-disc1.iso |
也可以通过浏览器，输入
http://192.168.199.135:50070/dfshealth.jsp 查看其存储情况。我的一个输出如下：
3、故障测试。需要2个数据节点，先关闭一个数据节点，然后网hadoop创建目录和拷贝数据。完毕后启动被关闭的数据节点的服务，观察其运行情况。
四、杂项
如果数据节点有2个以上的分区或硬盘需要拿来存放数据，则需要修改配置文件core-site.xml,请看下面的例子：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.0.135:9000</value>
<description>The name of the default file system. Either the literal string "local" or a host:port for DFS.</description>
</property>
<name>fs.default.name</name>
<value>hdfs://192.168.0.135:9000</value>
<description>The name of the default file system. Either the literal string "local" or a host:port for DFS.</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop-m:9001</value>
<description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map an
d reduce task.</description>
</property>
<name>mapred.job.tracker</name>
<value>hadoop-m:9001</value>
<description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map an
d reduce task.</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value >/disk2/hadoop/filesystem/tmp,/disk3/hadoop/filesystem/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<name>hadoop.tmp.dir</name>
<value >/disk2/hadoop/filesystem/tmp,/disk3/hadoop/filesystem/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>/ disk2/hadoop/filesystem/name,/disk3/hadoop/filesystem/name</value>
<description>Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited
list of directories then the name table is replicated in all of the directories, for redundancy. </description>
</property>
<name>dfs.name.dir</name>
<value>/ disk2/hadoop/filesystem/name,/disk3/hadoop/filesystem/name</value>
<description>Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited
list of directories then the name table is replicated in all of the directories, for redundancy. </description>
</property>
<property>
<name>dfs.data.dir</name>
<value >/disk2/hadoop/filesystem/data,/disk3/hadoop/filesystem/data</value>
<description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list
of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are
ignored.</description>
</property>
<name>dfs.data.dir</name>
<value >/disk2/hadoop/filesystem/data,/disk3/hadoop/filesystem/data</value>
<description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list
of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are
ignored.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication. The actual number of replications can be specified when the file is created. The default i
s used if replication is not specified in create time.</description>
</property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication. The actual number of replications can be specified when the file is created. The default i
s used if replication is not specified in create time.</description>
</property>
</configuration>
注意：<value>......</value>出现2个字段值，中间用逗号分割。如有更多共享分区，则依照这个规则加入。
本文转自sery51CTO博客，原文链接：http://blog.51cto.com/sery/214271
，如需转载请自行联系原作者
4152
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
