参考:http://www.cnblogs.com/micrari/p/5716851.html
》ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
》brewinstall Hadoop
》 ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
》cat ~/.ssh/id_dsa.pub~/.ssh/authorized_keys
》ssh localhost
在/usr/local/Cellar/hadoop/2.7.3/libexec/etc/下
- core-site.xml文件
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>
- hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
- mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
-
yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
运行Hadoop样例程序
此部分内容主要参考自Setting up Hadoop 2.6 on Mac OS X Yosemite
- 进入Hadoop的目录,以mac系统为例目录为
/usr/local/Cellar/hadoop/2.7.2/libexec
然后格式化文件系统:
$ bin/hdfs namenode -format
- 启动NameNode和DataNode的守护进程。
$ sbin/start-dfs.sh
- 启动ResourceManager和NodeManager的守护进程。
$ sbin/start-yarn.sh
- 访问localhost:50070和localhost:8088测试是否正常。
-
创建hdfs目录:
$ bin/hdfs dfs -mkdir /user $ bin/hdfs dfs -mkdir /user/robin
- 拷贝一些文件到input目录:
$ bin/hdfs dfs -put etc/hadoop input
- 运行样例:
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+'
-
在localhost:50070中的Utilities标签下找到/user/robin目录,下载part-r-00000文件,可以看到其中内容如下所示:
4 dfs.class 4 dfs.audit.logger 3 dfs.server.namenode. 2 dfs.period 2 dfs.audit.log.maxfilesize 2 dfs.audit.log.maxbackupindex 1 dfsmetrics.log 1 dfsadmin 1 dfs.servers 1 dfs.replication 1 dfs.file
- 进入Hadoop的目录,以mac系统为例目录为