关闭

Hadoop2.6.0 mac 安装

319人阅读 评论(0) 收藏 举报
分类:
1. 单机模式
(1) ssh设置远程登录:
xiaolan:~ xiaolan$@ssh-keygen -t rsa -P ""
xiaolan:~ xiaolan$cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
登录测试 
xiaolan:~ xiaolan$ssh localhost
xiaolan:~ xiaolan$exit
(2) 安装jdk(忽略)
(3) 配置环境变量
xiaolan:~ xiaolan$vim ~/.bash_profile
输入内容:
export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export YARN_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$YARN_HOME/etc/hadoop
:wq
让环境变量生效
xiaolan:~ xiaolan$source ~/.bash_profile
(4) 配置hadoop
进入Applications/hadoop-2.6.0/etc/hadoop
xiaolan:~ xiaolan$vim hadoop-env.sh
输入内容:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export PATH=$PATH:/Users/xiaolan/Applications/hadoop-2.6.0/bin
:wq
让环境变量生效
xiaolan:~ xiaolan$source hadoop-env.sh
单机模式安装成功
(5) 测试
xiaolan:~ xiaolan$mkdir input
xiaolan:~ xiaolan$cp **/** input
注:随便拷贝点文件
xiaolan:~ xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output
xiaolan:~ xiaolan$cat output/*
注:成功看到结果
2. 伪分布模式
(1) 配置core-site.xml
<configuration></configuration>之间增加如下内容:
<property>

         <name>fs.default.name</name>

         <value>hdfs://localhost:9000</value>

</property>

(2) 配置yarn-site.xml

<configuration></configuration>之间增加如下内容: 

<property> 

<name>yarn.noCHdemanager.aux-services</name> 

<value>mapreduce_shuffle</value> 

</property> 

 <property> 

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value> 

</property>

(3) 创建和配置mapred-site.xml

默认/etc/hadoop文件夹下有mapred.xml.template文件,复制更改用户名mapred.xml

<configuration></configuration>之间增加如下内容: 

<property> 

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

(4) 配置hdfs-site.xml

指定主机上作为namenode和datanode的目录:

/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name

/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data

<configuration></configuration>之间增加如下内容:

<property>

<name>dfs.replication</name>

         <value>1</value>

     </property>

     <property>

         <name>dfs.namenode.name.dir</name>

         <value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name</value>

     </property>

     <property>

         <name>dfs.datanode.data.dir</name>

         <value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data</value>

</property>

(5) 格式化hdfs

xiaolan:~ xiaolan$hdfs namenode -format

(6) 启动hadoop

xiaolan:~ xiaolan$cd Applications/hadoop-2.6.0/sbin

xiaolan:sbin xiaolan$./start-dfs.sh

xiaolan:sbin xiaolan$./start-yarn.sh

浏览器打开 http://localhost:50070/,会看到hdfs管理页面

浏览器打开 http://localhost:8088/,会看到hadoop进程管理页面

(7) WordCount验证

创建input目录

xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -mkdir input

fs:hadoop下的文件命令,和普通命令基本一致

拷贝文件

xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -copyFromLocal README.txt input

xiaolan:hadoop-2.6.0 xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output

xiaolan:hadoop-2.6.0 xiaolan$cat output/*

注:成功看到结果


linux:图文教程,版本较老:
http://www.linuxidc.com/Linux/2013-01/77681p4.htm
http://www.cnblogs.com/kinglau/p/3794433.html
官网安装教程:http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html

Eclipse配置:
http://andy-ghg.iteye.com/blog/1165453

注:
据说可以忽略:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable:

http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos


转帖:http://wollt1992.blog.163.com/blog/static/57013456201411244394378

0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:1044316次
    • 积分:12087
    • 等级:
    • 排名:第1200名
    • 原创:28篇
    • 转载:873篇
    • 译文:1篇
    • 评论:89条
    最新评论