一、配置ssh,无密码登录
1、mac开启ssh
2. mac上已经ssh了,在终端输入ssh-keygen -t rsa命令,碰到需要输入密码的直接按enter健即可。以rsa算法,生成公钥、私钥对。出现如下成功
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /Users/jia/.ssh/id_rsa. Your public key has been saved in /Users/jia/.ssh/id_rsa.pub. The key fingerprint is: d4:85:aa:83:ae:db:50:48:0c:5b:dd:80:bb:fa:26:a7 jia@JIAS-MacBook-Pro.local The key's randomart image is: +--[ RSA 2048]----+ |. .o.o .. | | =. . . ... | |. o. ... | | ... .. | | .... .S | | ... o | | ... . | |o oo. | |E*+o. | +-----------------+ |
在终端输入cd .ssh 进入.ssh目录,输入命令。
cp id_rsa.pub authorized_keys
导入公钥,即可。
二、配置文件
所有配置文件,均在$HADOOP_HOME/etc/hadoop目录下
(1) hadoop-env.sh
添加以下信息:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
export HADOOP_HOME=/Users/ycc/workspace/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin
export YARN_HOME=/Users/ycc/workspace/hadoop-2.6.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$YARN_HOME/etc/hadoop
(2) core-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
(3) yarn-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>yarn.noCHdemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
(4)mapred-site.xml
默认/etc/hadoop文件夹下有mapred-site.xml.template文件,复制更改用户名mapred.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
(5) hdfs-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file: /Users/ycc/workspace/hadoop-2.6.0/hdfs/data</value>
</property>
三、运行Hadoop
可以开始测试了:
1.先格式化hdfs
bin/hdfsnamenode –format
2、启动dfs、yarn
sbin/start-dfs.sh
sbin/start-yarn.sh
然后用jps查看java进程,应该能看到以下几个进程:
25361NodeManager
24931 DataNode
25258 ResourceManager
24797 NameNode
25098 SecondaryNameNode
还可以用以下命令查看hdfs的报告:
bin/hdfsdfsadmin -report 正常情况下可以看到以下内容
ConfiguredCapacity: 48228589568 (44.92 GB)
Present Capacity: 36589916160 (34.08 GB)
DFS Remaining: 36589867008 (34.08 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (1):
Name: 127.0.0.1:50010 (localhost)
Hostname: dc191
Decommission Status : Normal
Configured Capacity: 48228589568 (44.92 GB)
DFS Used: 49152 (48 KB)
Non DFS Used: 11638673408 (10.84 GB)
DFS Remaining: 36589867008 (34.08 GB)
DFS Used%: 0.00%
DFS Remaining%: 75.87%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue May 05 17:42:54 CST 2015
3、web管理界面查看
http://localhost:50070/
http://localhost:8088/
4、在hdfs的根目录下创建目录x
bin/hdfs dfs-mkdir /x
这样就在hdfs中创建了一个目录x
5、向hdfs中放入文件
bin/hdfs dfs-put README.txt /x
上面的命令会把当前目录下的README.TXT放入hdfs的/x目录中,在web管理界面里也可以看到该文件
6、WordCount验证
bin/hdfs dfs-mkdir /input
bin/hdfs dfs-put README.txt /input
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input/output
四、参考:
mac:http://wollt1992.blog.163.com/blog/static/57013456201411244394378
linux:图文教程,版本较老:
http://www.linuxidc.com/Linux/2013-01/77681p4.htm
http://www.cnblogs.com/kinglau/p/3794433.html
官网安装教程:
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html
注:
据说可以忽略:WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable:
http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos