1.下载hadoop
http://hadoop.apache.org/releases.html
在这个网站可以下载对应的版本。
2.将压缩包hadoop-2.8.1.tar.gz解压到相关目录下,
我解压的位置是
/Users/Fred/Downloads/hadoop-2.8.1
3.打开终端运行命令
java –version查看java安装情况及版本信息
如果没有安装Java可以到http://support.apple.com/kb/dl1421
下载安装
4.配置mac os环境
主要是ssh环境配置,在终端运行sshlocalhost
出现错误提示,说明当前用户没有权限。需要进入“系统偏好设置”中,进行设置
进入system preference --> sharing --> 勾选remotelogin,并设置allow accessfor all users。再次输入“sshlocalhost",再输入密码并确认之后,可以看到ssh成功。
-----------下面的内容没有进行设置,但是可以参考----------
不过这里面还有一个麻烦,就是每次都会要求输入用户密码。《Hadoop实战》提供了一种免登陆的方法。首先,输入
ssh-keygen -t dsa-P '' -f ~/.ssh/id_dsa
ssh-keygen表示生成秘钥;-t表示秘钥类型;-P用于提供密语;-f指定生成的秘钥文件。这个命令在”~/.ssh/“文件夹下创建两个文件id_dsa和id_dsa.pub,是ssh的一对儿私钥和公钥。接下来,将公钥追加到授权的key中去,输入:
cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
----------------参考内容----------------
5.设置环境变量
在实际启动hadoop之前需要配置三个文件,并且在配置之前需要进行环境变量的设置
exportHADOOP_HOME=/Users/Fred/Downloads/hadoop-2.8.1
exportPATH=$PATH:$HADOOP_HOME/bin
配置hadoop-env.sh,在
/Users/Fred/Downloads/hadoop-2.8.1/etc/hadoop目录下找到hadoop-env.sh文件,配置对应的内容
exportJAVA_HOME=${JAVA_HOME}
exportHADOOP_HEAPSIZE=2000
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
去掉前面的#号,或是添加上相应的数值;
注意第三个配置在OS X上最好进行配置,否则会报“Unable to load realm info from SCDynamicStore”。
6.接着对/Users/Fred/Downloads/hadoop-2.8.1/etc/hadoop目录下的三个文件进行配置
core-site.xml是全局配置文件,
hdfs-site.xml是HDFS的配置文件,
mapred-site.xml是MapReduce的配置文件
相关的配置信息如下:
core-site.xml
hdfs-site.xml
mapred-site.xml
7.格式化HDFS文件系统
在终端中,进行hadoop目录中的bin目录下
执行命令:./hadoop namenode –format
将会看到如下信息:
-----------终端信息展示-------------
FredMacBookPro:binFred$ ./hadoop namenode -format
DEPRECATED:Use of this script to execute hdfs command is deprecated.
Insteaduse the hdfs command for it.
17/10/1513:50:35 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG:Starting NameNode
STARTUP_MSG: user = Fred
STARTUP_MSG: host = fredmacbookpro.local/192.168.3.16
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.8.1
STARTUP_MSG: classpath =/users/Fred/Downloads/hadoop-2.8.1/etc/hadoop:/users/Fred/Downloads/hadoop-2.8.1/share/hadoop/common/lib/activation-1.1.jar:/users/Fred/Downloads/hadoop-2.8.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/users/Fred/Downloads/hadoop-2.8.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/users/Fred/Downloads/hadoop-2.8.1/share/hadoop/common/lib/
************************************************************/
17/10/1513:50:35 INFO namenode.NameNode: registered UNIX signal handlers for [TERM,HUP, INT]
17/10/1513:50:35 INFO namenode.NameNode: createNameNode [-format]
Javaconfig name: null
17/10/1513:50:37 INFO namenode.FSImageFormatProtobuf: Image file/users/Fred/Downloads/hadoop-2.8.1/tmp/hadoop-Fred/dfs/name/current/fsimage.ckpt_0000000000000000000of size 321 bytes saved in 0 seconds.
17/10/1513:50:37 INFO namenode.NNStorageRetentionManager: Going to retain 1 images withtxid >= 0
17/10/1513:50:37 INFO util.ExitUtil: Exiting with status 0
17/10/1513:50:37 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG:Shutting down NameNode at fredmacbookpro.local/192.168.3.16
************************************************************/
FredMacBookPro:binFred$ ls
--------------终端信息展示-----------------
8.验证安装结果
FredMacBookPro:binFred$ ./hadoop version
Hadoop2.8.1
Subversionhttps://git-wip-us.apache.org/repos/asf/hadoop.git -r20fe5304904fc2f5a18053c389e43cd26f7a70fe
Compiledby vinodkv on 2017-06-02T06:14Z
Compiledwith protoc 2.5.0
Fromsource with checksum 60125541c2b3e266cbf3becc5bda666
Thiscommand was run using/Users/Fred/Downloads/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1.jar
9.调用测试例子
FredMacBookPro:binFred$ ./hadoop jar ../share/hadoop/common/hadoop-common-2.8.1.jar
RunJar jarFile[mainClass] args...
FredMacBookPro:binFred$ ./hadoop jar../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 10 100
Numberof Maps = 10
Samplesper Map = 100
Javaconfig name: null
Nativeconfig name: /etc/krb5.conf
17/10/1515:54:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library foryour platform... using builtin-java classes where applicable
Wroteinput for Map #0
Wroteinput for Map #1
Wroteinput for Map #2
Wroteinput for Map #3
Wroteinput for Map #4
Wroteinput for Map #5
Wroteinput for Map #6
Wroteinput for Map #7
Wroteinput for Map #8
Wroteinput for Map #9
StartingJob
17/10/1515:54:12 INFO Configuration.deprecation: session.id is deprecated. Instead, usedfs.metrics.session-id
单机和单机伪分布式Hadoop系统安装完毕。