mac 下hadoop安装并运行例子

1 安装

＃brew install hadoop

安装的是2.6.0，目录为/usr/local/Cellar/hadoop,如果想安装其他版本，则下载tar包解压即可。地址：http://mirrors.cnnic.cn/apache/hadoop/common/
2 配置
将hadoop可执行路径bin和sbin都配置到环境变量中

export HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/home
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/sbin/:$HADOOP_HOME/bin/:$PATH

接下来配置伪分布式：
（1）
配置core－site.xml

➜  hadoop  cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

（2）hdfs的节点备份设置为1
配置hdfs-site.xml

➜  hadoop  cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

（3）配置mapred-site.xml

➜  hadoop  cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>hdfs://localhost:9000/</value>
  </property>
</configuration>

（4）将hadoop-env.sh里jdk路径配置好即可。
然后执行：格式化namenode

➜  hadoop namenode －format
➜  hadoop  start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/03/29 22:35:41 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 7082. Stop it first.
localhost: datanode running as process 7163. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 7266. Stop it first.
15/03/29 22:35:47 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 7377. Stop it first.
localhost: nodemanager running as process 7466. Stop it first.
➜  hadoop  jps
7163 DataNode
9619 Jps
7466 NodeManager
7377 ResourceManager
7082 NameNode
7266 SecondaryNameNode
➜  hadoop

可以看到五个节点都启动了。
3 运行例子wordcount
先创建目录input 以及文件并上传到hdfs

➜  libexec  ls input
file1.txt file2.txt
 ➜libexec hadoop fs -put input/file*.txt  /input
➜  libexec  hadoop fs -ls /input
15/03/29 22:46:37 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:46:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 shenyb supergroup         35 2015-03-29 11:02 /input/file1.txt
-rw-r--r--   1 shenyb supergroup         38 2015-03-29 11:02 /input/file2.txt
➜  libexec  hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output
运行如果不出错，结果输出到output目录如下：
➜  libexec  hadoop fs -ls /output
15/03/29 22:56:29 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 shenyb supergroup          0 2015-03-29 12:14 /output/_SUCCESS
-rw-r--r--   1 shenyb supergroup         67 2015-03-29 12:14 /output/part-r-00000
➜  libexec  hadoop fs -cat /output/part-r-00000
15/03/29 22:56:50 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ai  2
am  3
am2 1
are 2
i   2
man 1
man2    1
shenya  2
who 2
you 1
you2    1
➜  libexec