Hadoop学习（二） — Hadoop2.4伪分布式的安装与测试

一、Linux环境装备

1.安装jdk1.7以上版本

2.修改主机名

[hadoop123@tracker-wjw ~]$ vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=tracker-wjw

3.修改主机名和IP的映射关系，注意hadoop的安装中主机名不要有"_"

[hadoop123@tracker-wjw ~]$ vi /etc/hosts
127.0.0.1       tracker-wjw localhost   wang-Linux
192.168.5.130 wang-Linux
192.168.5.130 tracker-wjw

4.关闭防火墙

[hadoop123@tracker-wjw ~]$ chkconfig iptables --list
iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off

5.创建普通用户hadoop，赋予sudo权限，在普通用户的权限下安装hadoop

二、安装Hadoop

1.添加java_home

[hadoop123@tracker-wjw hadoop]$ vi hadoop-env.sh 

 26 # The java implementation to use.
 27 export JAVA_HOME=/usr/java/jdk1.7.0_79

2.data目录要以普通权限创建

[hadoop123@tracker-wjw hadoop]$ vi core-site.xml 


<configuration>

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://tracker-wjw:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop123/hadoop/hadoop-2.4.1/data</value>
</property>

</configuration>

3.

[hadoop123@tracker-wjw hadoop]$ vi mapred-site.xml 


<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

4.

[hadoop123@tracker-wjw hadoop]$ vi yarn-site.xml 

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>tracker-wjw</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

5.

[hadoop123@tracker-wjw hadoop]$ vi hdfs-site.xml 

<configuration>
<property>

<name>dfs.replication</name>
<value>1</value>

</property>
</configuration>

6.将hadoop环境添加到/etc/profile

export JAVA_HOME=/usr/local/java/jdk1.7.0_65
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME=/home/hadoop123/app/hadoop-2.4.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

7.格式化namenode

hadoop namenode -format

8.启动hadoop

[hadoop123@tracker-wjw sbin]$ start-dfs.sh
Starting namenodes on [tracker-wjw]
The authenticity of host 'tracker-wjw (127.0.0.1)' can't be established.
RSA key fingerprint is 79:dd:df:98:13:31:ee:88:5e:ff:fd:09:6c:81:7b:93.
Are you sure you want to continue connecting (yes/no)? yes
tracker-wjw: Warning: Permanently added 'tracker-wjw' (RSA) to the list of known hosts.
hadoop123@tracker-wjw's password: 
tracker-wjw: starting namenode, logging to /home/hadoop123/hadoop/hadoop-2.4.1/logs/hadoop-hadoop123-namenode-tracker-wjw.out
hadoop123@localhost's password: 
localhost: starting datanode, logging to /home/hadoop123/hadoop/hadoop-2.4.1/logs/hadoop-hadoop123-datanode-tracker-wjw.out
Starting secondary namenodes [0.0.0.0]
hadoop123@0.0.0.0's password: 
0.0.0.0: starting secondarynamenode, logging to /home/hadoop123/hadoop/hadoop-2.4.1/logs/hadoop-hadoop123-secondarynamenode-tracker-wjw.out
[hadoop123@tracker-wjw sbin]$ jps
2314 NameNode
2568 SecondaryNameNode
2670 Jps
2426 DataNode

9.启动yarn

[hadoop123@tracker-wjw sbin]$ start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/hadoop123/hadoop/hadoop-2.4.1/logs/yarn-hadoop123-resourcemanager-tracker-wjw.out
hadoop123@localhost's password: 
localhost: starting nodemanager, logging to /home/hadoop123/hadoop/hadoop-2.4.1/logs/yarn-hadoop123-nodemanager-tracker-wjw.out
[hadoop123@tracker-wjw sbin]$ jps
3114 Jps
3084 NodeManager
2314 NameNode
2568 SecondaryNameNode
2995 ResourceManager
2426 DataNode

三、测试

1.访问

在window下host配置主机名

2.测试上传、下载

[hadoop123@tracker-wjw ~]$ hadoop fs -put hadoop-2.4.1-x64.tar.gz  hdfs://tracker-wjw:9000/

[hadoop123@tracker-wjw ~]$ rm -rf hadoop-2.4.1-x64.tar.gz 
[hadoop123@tracker-wjw ~]$ ls
app  hadoop
[hadoop123@tracker-wjw ~]$ hadoop fs -get  hdfs://tracker-wjw:9000/hadoop-2.4.1-x64.tar.gz
[hadoop123@tracker-wjw ~]$ ls
app  hadoop  hadoop-2.4.1-x64.tar.gz

3.测试mapreduce

[hadoop123@tracker-wjw mapreduce]$ pwd
/home/hadoop123/hadoop/hadoop-2.4.1/share/hadoop/mapreduce

使用mapreduce的例子统计单词个数


[hadoop123@tracker-wjw mapreduce]$ hadoop fs -mkdir /wordcount
[hadoop123@tracker-wjw mapreduce]$ hadoop fs -mkdir /wordcount/test
[hadoop123@tracker-wjw mapreduce]$ hadoop fs -put  quangle.txt /wordcount/test

[hadoop123@tracker-wjw mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.4.1.jar wordcount /wordcount/test /wordcount/result

[hadoop123@tracker-wjw mapreduce]$ hadoop fs -ls /wordcount/result
[hadoop123@tracker-wjw mapreduce]$  hadoop fs -cat /wordcount/result/part-r-00000
Hadoop  1
grils   1
hello   2
is      1
magical 1
whoami  1
world   2

安装成功

四、添加ssh无密登录

[hadoop123@tracker-wjw ~]$ ssh-keygen -t rsa （四个回车）

会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥），将公钥拷贝到要免登陆的机器上

[hadoop123@tracker-wjw ~]$ ssh-copy-id localhost

或

 
 
exit                           # 退出刚才的 ssh localhost
cd ~/.ssh/                     # 若没有该目录，请先执行一次ssh localhost
ssh-keygen -t rsa              # 会有提示，都按回车就可以
cat ./id_rsa.pub >> ./authorized_keys  # 加入授权