hadoop

下载：http://mirrors.cnnic.cn/apache/hadoop/common/

   http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-1.2.1/

 

安装：http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html

 

mr  例子：http://hi.baidu.com/whyang2006/item/436a720c4e15a013addc70c4

 

主启动、停止：

$ start-all.sh

$ stop-all.sh

[root@Master ~]# su - hadoop
-bash-3.2$ /usr/hadoop/bin/start-all.sh 
starting namenode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-namenode-Master.Hadoop.out
192.168.1.101: starting datanode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-datanode-Slave1.Hadoop.out
192.168.1.100: starting secondarynamenode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-Master.Hadoop.out
starting jobtracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-Master.Hadoop.out
192.168.1.101: starting tasktracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-Slave1.Hadoop.out

 

安装成功测试：

主服务器：

-bash-3.2$ jps

10030 JobTracker

9790 NameNode

10143 Jps

9949 SecondaryNameNode

从服务器

-bash-3.2$ jps

11579 TaskTracker

11501 DataNode

11660 Jps

 

问题1：不能启动从节点

2013-12-24 00:49:01,198 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local directories are writable

2013-12-24 00:48:59,819 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: can not create directory: /usr/hadoop/tmp/dfs/data

解决：chown -R hadoop:hadoop  hadoop

 

ssh无密传输：

su - hadoop

生成非对称加密的私钥和公钥

ssh-keygen -t rsa -P ''

一路回车，默认值。

运行命令后会再/home/hadoop 目录下生成.ssh的隐藏文件夹，运行ll -a 查看是否生成成功

[hadoop@Master ~]$ ssh-keygen -t rsa -P ''

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 

Created directory '/home/hadoop/.ssh'.

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

c0:e3:2e:a5:38:a3:0d:25:0a:50:93:57:28:4d:b9:60 hadoop@Master.Hadoop

[hadoop@Master ~]$ ll -a

total 16

drwx------ 3 hadoop hadoop 4096 Dec 19 07:44 .

drwxr-xr-x 4 root   root   4096 Aug 16  2012 ..

drwx------ 2 hadoop hadoop 4096 Dec 19 07:44 .ssh

 

修改授权文件的权限

chmod 600 ~/.ssh/authorized_keys

切换到root用户修改sshd_config文件，使ssh支持无密码登录

su - root

vi /etc/ssh/sshd_config

去掉注释

RSAAuthentication yes # 启用 RSA 认证

PubkeyAuthentication yes # 启用公钥私钥配对认证方式

AuthorizedKeysFile .ssh/authorized_keys #公钥文件路径(和上面生成的文件同

 

将master节点上的公钥复制到slave节点的/home/hadoop目录下

scp ~/.ssh/id_rsa.pub hadoop@192.168.1.101:~/

hadoop用户连接slave机器

创建.ssh的隐藏文件夹

mkdir ~/.ssh

修改权限

chmod 700 ~/.ssh

将master的公钥信息加入到slave的授权文件中

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

完成从slave ssh登录master的过程，master同样需要slave的公钥id_rsa.pub

 测试ssh成功：

ssh 192.168.1.101

如果没有提示输入密码则无密码登录设置成功否则设置失败

  

pig hive   http://zhidao.baidu.com/link?

url=6N3qTl2WrtMZ6cl1sPekVrTjdFxtSBTGvQGEuMwuEK0WmAVWhCVmayS-JzfkZiu0GqFWm1WKXXdrsPmHfQVttK

# hadoop job -list
0 jobs currently running
JobId   State   StartTime       UserName        Priority        SchedulingInfo

 --------------

参考： http://blog.csdn.net/yy_done/article/details/7006489
cp /usr/hadoop/src/examples/org/apache/hadoop/examples/WordCount.java ~
mkdir classes
javac -classpath /usr/hadoop/hadoop-core-1.2.1.jar:/usr/hadoop/lib/commons-cli-1.2.jar -d classes WordCount.java 
jar -cvf wordcount.jar -C classes/ .
hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output 

-bash-3.2$ hadoop fs -put file01 input
-bash-3.2$ hadoop fs -put file02 input
-bash-3.2$ hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output
14/02/23 23:28:09 INFO input.FileInputFormat: Total input paths to process : 2
14/02/23 23:28:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/02/23 23:28:09 WARN snappy.LoadSnappy: Snappy native library not loaded
14/02/23 23:28:10 INFO mapred.JobClient: Running job: job_201402232134_0015
14/02/23 23:28:11 INFO mapred.JobClient:  map 0% reduce 0%
14/02/23 23:28:29 INFO mapred.JobClient:  map 100% reduce 0%
14/02/23 23:28:37 INFO mapred.JobClient:  map 100% reduce 33%
14/02/23 23:28:38 INFO mapred.JobClient:  map 100% reduce 100%
14/02/23 23:28:40 INFO mapred.JobClient: Job complete: job_201402232134_0015
14/02/23 23:28:40 INFO mapred.JobClient: Counters: 29
14/02/23 23:28:40 INFO mapred.JobClient:   Job Counters
14/02/23 23:28:40 INFO mapred.JobClient:     Launched reduce tasks=1
14/02/23 23:28:40 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=33728
14/02/23 23:28:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/02/23 23:28:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/02/23 23:28:40 INFO mapred.JobClient:     Launched map tasks=2
14/02/23 23:28:40 INFO mapred.JobClient:     Data-local map tasks=2
14/02/23 23:28:40 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9437
14/02/23 23:28:40 INFO mapred.JobClient:   File Output Format Counters
14/02/23 23:28:40 INFO mapred.JobClient:     Bytes Written=33
14/02/23 23:28:40 INFO mapred.JobClient:   FileSystemCounters
14/02/23 23:28:40 INFO mapred.JobClient:     FILE_BYTES_READ=55
14/02/23 23:28:40 INFO mapred.JobClient:     HDFS_BYTES_READ=268
14/02/23 23:28:40 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=164949
14/02/23 23:28:40 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=33
14/02/23 23:28:40 INFO mapred.JobClient:   File Input Format Counters
14/02/23 23:28:40 INFO mapred.JobClient:     Bytes Read=38
14/02/23 23:28:40 INFO mapred.JobClient:   Map-Reduce Framework
14/02/23 23:28:40 INFO mapred.JobClient:     Map output materialized bytes=61
14/02/23 23:28:40 INFO mapred.JobClient:     Map input records=2
14/02/23 23:28:40 INFO mapred.JobClient:     Reduce shuffle bytes=61
14/02/23 23:28:40 INFO mapred.JobClient:     Spilled Records=8
14/02/23 23:28:40 INFO mapred.JobClient:     Map output bytes=62
14/02/23 23:28:40 INFO mapred.JobClient:     CPU time spent (ms)=12970
14/02/23 23:28:40 INFO mapred.JobClient:     Total committed heap usage (bytes)=248127488
14/02/23 23:28:40 INFO mapred.JobClient:     Combine input records=6
14/02/23 23:28:40 INFO mapred.JobClient:     SPLIT_RAW_BYTES=230
14/02/23 23:28:40 INFO mapred.JobClient:     Reduce input records=4
14/02/23 23:28:40 INFO mapred.JobClient:     Reduce input groups=4
14/02/23 23:28:40 INFO mapred.JobClient:     Combine output records=4
14/02/23 23:28:40 INFO mapred.JobClient:     Physical memory (bytes) snapshot=327409664
14/02/23 23:28:40 INFO mapred.JobClient:     Reduce output records=4
14/02/23 23:28:40 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1123270656
14/02/23 23:28:40 INFO mapred.JobClient:     Map output records=6
-bash-3.2$ hadoop fs -cat output/part-r-00000 
Bye     1
Goodbye 1
Hadoop  2
World   2

-------------

安装插件：把hadoop-eclipse-plugin-1.2.1.jar  放到eclipse\plugins下，重启eclipse。

eclispe调试worldcount修改如下：

conf.set("mapred.job.tracker", "192.168.1.100:9001"); 
String[] ars = new String[] { "input","output" };
String[] otherArgs = new GenericOptionsParser(conf, ars).getRemainingArgs();

  

eclipse运行报错：ClassNotFoundException: org.apache.hadoop.examples.WordCount$IntSumReducer
解决：eclipse4.2 换位3.7,  hadoop-eclipse-plugin-1.2.1.jar 插件和eclipse4.2不兼容的问题。

-----

查看日志：

more /usr/hadoop/logs/hadoop-hadoop-namenode-Master.Hadoop.log

tail -200  /usr/hadoop/logs/hadoop-hadoop-namenode-Master.Hadoop.log

tail -200  /usr/hadoop/logs/hadoop-hadoop-datanode-Slave1.Hadoop.log

问题1:java.io.IOException: NameNode is not formatted.

解决：hadoop namenode -format

注意：Re-format filesystem in /usr/hadoop/tmp/dfs/name ? (Y or N) Y    注意输入大写Y,小写不行。

问题2：ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /usr/hadoop/tmp/dfs/data: namenode namespaceID = 2053604871; datanode namespaceID = 1352668130

解决：namespaceID 不一致问题。修改了datanode的/usr/hadoop/tmp/dfs/data/current的VERSION文件的namespaceID=2053604871

问题3：java.io.FileNotFoundException: /usr/hadoop/tmp/dfs/name/current/VERSION (Permission denied)

解决：不小心用了root管理员账号启动了start-all.sh，后来取消了，但是还是在/home/hadoop/tmp/生成了一些所有者为root的文件。将文件全部删除 或者改为hadoop用户 # chown -R hadoop:hadoop /usr/hadoop/tmp