结合网上多份文档,不断反复的修正hadoop启动和运行过程中出现的问题,终于把Hadoop2.5.2伪分布式安装起来,跑通了wordcount例子。Hadoop的安装复杂性的体现之一是,Hadoop的安装文档非常多,但是能一个文档走下来的少之又少,尤其是Hadoop不同版本的配置差异非常的大。Hadoop2.5.2于前两天发布,但是它的配置跟2.5.0,2.5.1没有分别。
系统环境: Ubuntu 12.04 LTS x86_32
一、创建用户组和用户
- 创建用户组,为系统添加一个用户组hadoop
sudo addgroup hadoop
- 创建用户,为系统添加一个用户hadoop
useradd -g hadoop hadoop
- 使用hadoop用户登陆
su hadoop
二、使用SSH免密码登录
- 执行如下命令,生成ssh的密钥,创建的key的路径是/home/hadoop/.ssh/id_rsa
ssh-keygen -t rsa -P ""
- 将 /home/hadoop/.ssh/id_rsa.pub中的内容追加到/home/hadoop/.ssh/authorized_keys中,保存
- 执行ssh localhost,验证无密码即可登陆
三、禁用IPv6
- 执行如下命令查看当前IPv6是否禁用,1表示禁用,0表示未禁用,默认是0
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
- 编辑如下文件,添加三行,禁用IPv6
sudo vim /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
- 重启机器,再次查看IPv6是否禁用
四、安装配置JDK
- 编辑/etc/profile文件,设置JAVA相关的系统变量
export JAVA_HOME=/software/devsoftware/jdk1.7.0_55
export PATH=$JAVA_HOME/bin:$PATH
五、安装配置Hadoop2.5.2
- 编辑/etc/profile文件,设置Hadoop相关的系统变量
export HADOOP_HOME=/home/hadoop/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$PATH
- 执行如下使上面配置的系统变量生效
source /etc/profile
- 将JDK设置到Hadoop的环境脚本/home/hadoop/hadoop-2.5.2/etc/hadoop/hadoop-env.sh中,追加一行
export JAVA_HOME=/software/devsoftware/jdk1.7.0_55
六、Hadoop2.5.2配置文件设置
Hadoop2.5.2有四个配置文件需要配置,它们都位于/home/hadoop/hadoop-2.5.2/etc/hadoop目录下。四个文件分别是
- core-site.xml
- yarn-site.xml
- mapred-site.xml
- hdfs-site.xml
这写配置文件中有些需要手工创建目录,有些需要根据系统的实际情况,设置hostname,hostname不能是IP或者localhost,需要在/etc/hosts中进行设置。需要补充一点,有几个文档指出,127.0.0.1最好只跟一个hostname(即Hadoop用到的)绑定,把其余的注释掉。这个究竟是否产生影响,没有测,只是按照网上的说法,只保留一个hostname
6.1 core-site.xml配置
<configuration> <property> <name>hadoop.tmp.dir</name> <!--目录必须手动创建出来--> <value>/home/hadoop/data/tmp</value> <description>A base for other temporary directories.</description> </property> <!--file system properties--> <property> <name>fs.defaultFS</name> <!--HDFS的服务地址,只能使用域名,不能设置为IP或者localhost--> <value>hdfs://hostname:9000</value> </property> <property> <!--使用Hadoop自带的so库--> <name>hadoop.native.lib</name> <value>true</value> <description>Should native hadoop libraries, if present, be used.</description> </property> </configuration>
6.2 mapred-site.xml配置
mapred-site.xml文件默认不存在,使用cp命令从
mapred-site.xml.template拷贝一份
cp mapred-site.xml.template mapred-site.xml
做如下设置,
<configuration> <property> <name>mapreduce.framework.name</name> <!--yarn全是小写,不是Yarn--> <value>yarn</value> </property> </configuration>
6.3 yarn-site.xml配置
<configuration> <!-- Site specific YARN configuration properties --> <property> <!--yarn是小写,或许大些Y也可以--> <name>yarn.nodemanager.aux-services</name> <!--不是mapreduce.shuffle--> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <description>The address of the applications manager interface in the RM.</description> <name>Yarn.resourcemanager.address</name> <!--根据实际情况,设置hostname域名--> <value>hostname:18040</value> </property> <property> <description>The address of the scheduler interface.</description> <name>Yarn.resourcemanager.scheduler.address</name> <!--根据实际情况,设置hostname域名--> <value>hostname:18030</value> </property> <property> <description>The address of the RM web application.</description> <name>Yarn.resourcemanager.webapp.address</name> <!--根据实际情况,设置hostname域名--> <value>hostname:18088</value> </property> <property> <description>The address of the resource tracker interface.</description> <name>Yarn.resourcemanager.resource-tracker.address</name> <!--根据实际情况,设置hostname域名--> <value>hostname:8025</value> </property> </configuration>
6.4 hdfs-site.xml 配置
<configuration> <property> <name>dfs.namenode.name.dir</name> <!--手工创建好--> <value>/home/hadoop/data/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <!--手工创建好--> <value>/home/hadoop/data/hdfs/data</value> </property> <property> <!--HDFS文件复本数--> <name>dfs.replication</name> <value>1</value> </property> </configuration>
七、Hadoop初始化并启动
- 格式化Hadoop NameNode
hadoop namenode -format
观察日志,如果有输出中包括Storage directory /home/hadoop/data/hdfs/name has been successfully formatted,则表示格式化成功
- 启动Hadoop
/home/hadoop/hadoop-2.5.2/sbin/start-all.sh
- 使用JDK的jps检查Hadoop状态,如果是如下结果,则表示安装成功
10682 DataNode
10463 NameNode
11229 ResourceManager
24647 Jps
11040 SecondaryNameNode
11455 NodeManager
- 使用netstat -anp|grep java观察Hadoop端口号使用情况
tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 11455/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 11040/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 10463/java
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:34456 0.0.0.0:* LISTEN 11455/java
tcp 0 0 0.0.0.0:13562 0.0.0.0:* LISTEN 11455/java
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 10682/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 10682/java
tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:8031 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:8032 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:8033 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 10682/java
tcp 0 0 0.0.0.0:8040 0.0.0.0:* LISTEN 11455/java
- 浏览NameNode、DataNode信息,可以查看HDFS状态信息
http://hostname:50070
- 浏览ResourceManagered运行状态,可以浏览MapReduce任务的执行情况
http://hostname:8088
八、运行Hadoop自带的WordCount实例
- 创建本地文件用于计算这个文件中的单词数
echo "My first hadoop example. Hello Hadoop in input. " > /home/hadoop/input
- 创建HDFS输入目录,用于将上面的文件写入这个目录
hadoop fs -mkdir /user/hadooper
- 传文件到HDFS输入目录
hadoop fs -put /home/hadoop/input /user/hadooper
- 执行Hadoop自带的WordCount例子
hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/hadooper/input /user/hadooper/output
- MapReduce的过程输出
hadoop@hostname:~/hadoop-2.5.2/share/hadoop/mapreduce$ hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/hadooper/input /user/hadooper/output
14/11/23 19:45:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/11/23 19:45:05 INFO input.FileInputFormat: Total input paths to process : 1
14/11/23 19:45:05 INFO mapreduce.JobSubmitter: number of splits:1
14/11/23 19:45:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1416742510596_0001
14/11/23 19:45:06 INFO impl.YarnClientImpl: Submitted application application_1416742510596_0001
14/11/23 19:45:07 INFO mapreduce.Job: The url to track the job: http://hostname:8088/proxy/application_1416742510596_0001/
14/11/23 19:45:07 INFO mapreduce.Job: Running job: job_1416742510596_0001
14/11/23 19:45:18 INFO mapreduce.Job: Job job_1416742510596_0001 running in uber mode : false
14/11/23 19:45:18 INFO mapreduce.Job: map 0% reduce 0%
14/11/23 19:45:26 INFO mapreduce.Job: map 100% reduce 0%
14/11/23 19:45:36 INFO mapreduce.Job: map 100% reduce 100%
14/11/23 19:45:37 INFO mapreduce.Job: Job job_1416742510596_0001 completed successfully
14/11/23 19:45:37 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=102
FILE: Number of bytes written=195793
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=168
HDFS: Number of bytes written=64
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5994
Total time spent by all reduces in occupied slots (ms)=6925
Total time spent by all map tasks (ms)=5994
Total time spent by all reduce tasks (ms)=6925
Total vcore-seconds taken by all map tasks=5994
Total vcore-seconds taken by all reduce tasks=6925
Total megabyte-seconds taken by all map tasks=6137856
Total megabyte-seconds taken by all reduce tasks=7091200
Map-Reduce Framework
Map input records=1
Map output records=8
Map output bytes=80
Map output materialized bytes=102
Input split bytes=119
Combine input records=8
Combine output records=8
Reduce input groups=8
Reduce shuffle bytes=102
Reduce input records=8
Reduce output records=8
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=101
CPU time spent (ms)=2640
Physical memory (bytes) snapshot=422895616
Virtual memory (bytes) snapshot=2055233536
Total committed heap usage (bytes)=308281344
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=49
File Output Format Counters
Bytes Written=64
- 查看MapReduce的运行结果
hadoop@hostname:~/hadoop-2.5.2/share/hadoop/mapreduce$ hadoop fs -cat /user/hadooper/output/part-r-00000
Hadoop 1
Hello 1
My 1
example. 1
first 1
hadoop 1
in 1
input. 1
九、运行Hadoop的PI程序
hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 10 10
执行结果是3.200000000000000000
十、Hadoop常见问题
1. hadoop不正常推出后,重启后,NameNode将进入Safe Mode,不能提交任务,解决办法:
hadoop dfsadmin -safemode leave