CentOS 6.4 安装伪分布式Hadoop 1.0.3

在本章中学习如何在CentOS 6.4上安装配置Hadoop开发环境。

[size=x-large][color=blue]配置ssh及生成密码对[/color][/size]
首先需要安装ssh：[quote]
[color=blue]# yum install ssh[/color]
[/quote]
紧接着启动ssh，并将之设置开机运行：[quote]
[color=blue]# service sshd start
# chkconfig sshd on[/color]
[/quote]
好，现在已经安装并启动了sshd，接下来我们来为hadoop生成面登录的密码对：[quote]
[color=blue]# ssh-keygen[/color]
[/quote]
然后接下来的交互中一路回车，如下图所示：
[img]http://dl2.iteye.com/upload/attachment/0086/4050/922de2b1-1d35-37ba-8cf6-794ce8579116.png[/img]
再然后将刚才生成的公钥copy到.ssh/authorized_keys目录下：[quote]
[color=blue]# cd /root/.ssh
# cat ./id_rsa.pub>>./authorized_keys[/color]
[/quote]
这样，ssh和免密码登录的配置就应经好了，见下图：
[img]http://dl2.iteye.com/upload/attachment/0086/4057/24f45e0d-285b-33d8-a43d-6d19653736c0.png[/img]

[size=large][color=blue]配置机器时间同步[/color][/size]
手动同步：[quote]
[color=blue]# usr/sbin/ntpdate cn.pool.ntp.org[/color]
[/quote]
自动同步：[quote]
[color=blue]# crontab e[/color]
[color=indigo]0 1 * * * /usr/sbin/ntpdate cn.pool.ntp.org[/color]
[/quote]
[img]http://dl2.iteye.com/upload/attachment/0086/4070/eac88315-9d7f-3512-a5d7-5dee9e7f1835.png[/img]
[img]http://dl2.iteye.com/upload/attachment/0086/4068/e7a8b9b1-4842-32e4-9e89-1d0e52713d4e.png[/img]

[size=large][color=blue]安装Hadoop 1.0.3[/color][/size]
在安装hadoop之前，需要先配置Java环境，java安装过程这里不再赘述。
下载并解压hadoop 1.0.3：[quote]
[color=blue]# wget https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3-bin.tar.gz
# chmod +x hadoop-1.0.3-bin.tar.gz
# tar xzvf hadoop-1.0.3-bin.tar.gz[/color]
[/quote]
然后将hadoop-1.0.3复制到/opt/bigdata目录下：[quote]
[color=blue]# cp -rf hadoop-1.0.3 /opt/bigdata[/color]
[/quote]
紧接着配置hadoop的环境：[quote]
[color=blue]# vi /ect/profile.d/java.sh [/color]
[/quote]
[img]http://dl2.iteye.com/upload/attachment/0086/4085/68c53649-cad7-3e7c-a6a2-827a805b9a9a.png[/img]
[quote]
# source /etc/profile
[/quote]
然后我们来看配置后的效果：
[img]http://dl2.iteye.com/upload/attachment/0086/4088/a802cf3b-b121-3336-8196-544679c473fc.png[/img]

最后，配置hadoop的一些相关配置文件：
1、配置$HADOOP_HOME/conf/hadoop-env.sh，在文件的末尾添加下面内容:[quote]
#set java environment
export JAVA_HOME=/usr/local/jdk6
[/quote]
2、配置core-site.xml文件
core-site.xml是Hadoop核心配置文件，这里配置的是HDFS的地址和端口号：[quote]
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/bigdata/hadoop-tmp</value>
</property>

<!-- file system properties -->
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>

<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>

[/quote]
如没有配置hadoop.tmp.dir参数，此时系统默认的临时目录为：/tmp/hadoo-hadoop。而这个目录在每次重启后都会被干掉，必须重新执行format才行，否则会出错
3、配置hdfs-site.xml文件
修改Hadoop中HDFS的配置，配置的备份方式默认为3:[quote]
<configuration>

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.block.size</name>
<value>134217728</value>
</property>


<configuration>
[/quote]
4、配置mapred-site.xml文件
配置的是JobTracker的地址和端口：[quote]
<configuration>

<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>

</configuration>
[/quote]
好了，到这里hadoop在CentOS 6.4下的伪分布式环境已经陪好了。
最后一步还要修改/etc/hosts，将用户名映射到127.0.0.1，如：[quote]
127.0.0.1 JackyChen
[/quote]
不然DataNode节点启动不起来。接下来然我们初始化NameNode:[quote]
[color=blue]# cd bin
# hadoop namenode -format[/color]
[/quote]
[img]http://dl2.iteye.com/upload/attachment/0086/6460/2c122550-d75d-34b2-87fd-fcb7c431cb9e.png[/img]
最后启动hadoop：[quote]
[color=blue]# ./start-all.sh[/color]
[/quote]
[img]http://dl2.iteye.com/upload/attachment/0086/6462/f8737d80-1272-3a1d-91a3-b03fbe006e0d.png[/img]

[img]http://dl2.iteye.com/upload/attachment/0086/6842/16ff3817-86ec-3cee-bf46-40566ffa801f.png[/img]

[img]http://dl2.iteye.com/upload/attachment/0086/6845/2f89ce85-f6bc-3408-a359-e484ea73ccce.png[/img]