hadoop2.2.0安装与启动

hadoop2.2.0安装与启动

Setup newest Hadoop 2.x (2.2.0) on Ubuntu

In this tutorial I am going to guide you through setting up hadoop 2.2.0 environment on Ubuntu.

Prerequistive

[java]  view plain copy

1. $ sudo apt-get install openjdk-7-jdk  
2. $ java -version  
3. java version "1.7.0_25"  
4. OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4ubuntu3)  
5. OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)  
6. $ cd /usr/lib/jvm  
7. $ ln -s java-7-openjdk-amd64 jdk  
8.   
9. $ sudo apt-get install openssh-server  

Add Hadoop Group and User

[java]  view plain copy

1. $ sudo addgroup hadoop  
2. $ sudo adduser --ingroup hadoop hduser  
3. $ sudo adduser hduser sudo  

After user is created, re-login into ubuntu using hduser

Setup SSH Certificate

[java]  view plain copy

1. $ ssh-keygen -t rsa -P ''  
2. ...  
3. Your identification has been saved in /home/hduser/.ssh/id_rsa.  
4. Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.  
5. ...  
6. $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  
7. $ ssh localhost  

Download Hadoop 2.2.0

[java]  view plain copy

1. $ cd ~  
2. $ wget http://www.trieuvan.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz  
3. $ sudo tar vxzf hadoop-2.2.0.tar.gz -C /usr/local  
4. $ cd /usr/local  
5. $ sudo mv hadoop-2.2.0 hadoop  
6. $ sudo chown -R hduser:hadoop hadoop  

Setup Hadoop Environment Variables

[java]  view plain copy

1. $cd ~  
2. $vi .bashrc  
3.   
4. paste following to the end of the file  
5.   
6. #Hadoop variables  
7. export JAVA_HOME=/usr/lib/jvm/jdk/  
8. export HADOOP_INSTALL=/usr/local/hadoop  
9. export PATH=$PATH:$HADOOP_INSTALL/bin  
10. export PATH=$PATH:$HADOOP_INSTALL/sbin  
11. export HADOOP_MAPRED_HOME=$HADOOP_INSTALL  
12. export HADOOP_COMMON_HOME=$HADOOP_INSTALL  
13. export HADOOP_HDFS_HOME=$HADOOP_INSTALL  
14. export YARN_HOME=$HADOOP_INSTALL  
15. ###end of paste  
16.   
17. $ cd /usr/local/hadoop/etc/hadoop  
18. $ vi hadoop-env.sh  
19.   
20. #modify JAVA_HOME  
21. export JAVA_HOME=/usr/lib/jvm/jdk/  

Re-login into Ubuntu using hdser and check hadoop version

[java]  view plain copy

1. $ hadoop version  
2. Hadoop 2.2.0  
3. Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768  
4. Compiled by hortonmu on 2013-10-07T06:28Z  
5. Compiled with protoc 2.5.0  
6. From source with checksum 79e53ce7994d1628b240f09af91e1af4  
7. This command was run using /usr/local/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar  

At this point, hadoop is installed.

Configure Hadoop

修改hadoop-env.sh文件，改变java_home的值

[java]  view plain copy

1. $ cd /usr/local/hadoop/etc/hadoop  
2. $ vi core-site.xml  
3. #Paste following between <configuration>  
4.   
5.   
6.    fs.default.name  
7.    hdfs://localhost:9000  
8.   
9.   
10.   
11. $ vi yarn-site.xml  
12. #Paste following between <configuration>  
13.   
14.   
15.    yarn.nodemanager.aux-services  
16.    mapreduce_shuffle  
17.   
18.   
19.    yarn.nodemanager.aux-services.mapreduce.shuffle.class  
20.    org.apache.hadoop.mapred.ShuffleHandler  
21.   
22.   
23.   
24. $ mv mapred-site.xml.template mapred-site.xml  
25. $ vi mapred-site.xml  
26. #Paste following between <configuration>  
27.   
28.   
29.    mapreduce.framework.name  
30.    yarn  
31.   
32.   
33.   
34. $ cd ~  
35. $ mkdir -p mydata/hdfs/namenode  
36. $ mkdir -p mydata/hdfs/datanode  
37. $ cd /usr/local/hadoop/etc/hadoop  
38. $ vi hdfs-site.xml  
39. Paste following between <configuration> tag  
40.   
41.   
42.    dfs.replication  
43.    1  
44.    
45.    
46.    dfs.namenode.name.dir  
47.    file:/home/hduser/mydata/hdfs/namenode  
48.    
49.    
50.    dfs.datanode.data.dir  
51.    file:/home/hduser/mydata/hdfs/datanode  
52.    

Format Namenode

[java]  view plain copy

1. hduser@ubuntu40:~$ hdfs namenode -format  

Start Hadoop Service

[java]  view plain copy

1. $ start-dfs.sh  
2. ....  
3. $ start-yarn.sh  
4. ....  
5.   
6. hduser@ubuntu40:~$ jps  
7. If everything is sucessful, you should see following services running  
8. 2583 DataNode  
9. 2970 ResourceManager  
10. 3461 Jps  
11. 3177 NodeManager  
12. 2361 NameNode  
13. 2840 SecondaryNameNode  

Run Hadoop Example

[java]  view plain copy

1. hduser@ubuntu: cd /usr/local/hadoop  
2. hduser@ubuntu:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5  
3.   
4. Number of Maps  = 2  
5. Samples per Map = 5  
6. 13/10/21 18:41:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
7. Wrote input for Map #0  
8. Wrote input for Map #1  
9. Starting Job  
10. 13/10/21 18:41:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032  
11. 13/10/21 18:41:04 INFO input.FileInputFormat: Total input paths to process : 2  
12. 13/10/21 18:41:04 INFO mapreduce.JobSubmitter: number of splits:2  
13. 13/10/21 18:41:04 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name  

运行结果是pi is 3.60000

1. ...  

[java]  view plain copy

1.    

[java]  view plain copy

1.    

[java]  view plain copy

1. 转帖地址：<a target="_blank" href="http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1">http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1</a>