hadoop搭建_在Mac环境中搭建Hadoop

THANK YOU 11.28

在Mac环境下搭建Hadoop，简单粗暴一键安装，包含过程中可能会出现的坑，大家也可以看看解决方案。

JAVA环境配置

、

jdk安装地址

https://www.oracle.com/java/technologies/javase-jdk15-downloads.html

(建议Chrome打开，有翻译功能，已安装或英语过关忽略)

查看jdk版本信息

1 ~ $ java -version

2 java version "1.8.0_211"

3 Java(TM) SE Runtime Environment (build 1.8.0_211-b12)

4 Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

(版本信息可能不一样，根据自己安装情况)

查看JAVA的安装位置信息

1 ~ $ /usr/libexec/java_home 

2 /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home

SSH配置

生成密钥

.ssh $ ssh-keygenGenerating public/private rsa key pair.Enter file in which to save the key (/Users/lop-w/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /Users/lop-w/.ssh/id_rsa.Your public key has been saved in /Users/lop-w/.ssh/id_rsa.pub.The key fingerprint is:SHA256:pCpvY/JldgpCOQgpCO6ulPQWALC4bDDkmG7e0wZPvw0 lop-w@liuyuanchengdeMacBook-Pro.localThe key's randomart image is:+---[RSA 3072]----+|=.               ||X+               ||%+      .        ||B+..   o         ||oB+o .. S        ||*.+.*..          || =o=o++E.        ||o o+*= o+        ||.  =o... .       |+----[SHA256]-----+ cd ~/.ssh ～ $ ls -l  

配置ssh免密登陆(决定着后面hadoop服务开启正常)

开启在Mac上的允许远程登录：->系统偏好设置->共享->如下图

将密钥文件输入到服务器中使用ssh-copy-id脚本工具

.ssh $ ssh-copy-id -i [公钥文件]user@hostThe authenticity of host 'localhost (::1)' can't be established.ECDSA key fingerprint is SHA256:MulbiBXzPb1bZrC+Br1nJcmSoq6yagAfGHh3QVnHKxw.Are you sure you want to continue connecting (yes/no/[fingerprint])? yes/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysPassword:Number of key(s) added:        1Now try logging into the machine, with:   "ssh 'lop-w@localhost'"and check to make sure that only the key(s) you wanted were added.

*过程中可能需要输入密码

*[公钥文件]=id__rsa.pub

*user=Mac的用户名

*host=localhost

2 .ssh $ ssh-add -K [私钥文件]

*[私钥文件]=id_rsa

.ssh $ chmod 600 ~/.ssh/authorized_keys~ $ ssh Mac用户名@localhost

(成功免密登陆)

****以上步骤完成即可免密登陆 如不能看下文****

(小编开始在这里很崩溃，因为一切就绪还是无法免密，痛苦之时，猛然想起是否权限问题)

cd ~~ $ chmod 700.ssh~ $ ls -l

* 查看你的用户名是不是755 drwxr-xr-x+ 如果不是755 则执行 chmod 755 用户名

这个时候就能够成功使用 ssh Mac用户名@localhost 免密登陆啦！

安装Hadoop

创建放置hadoop的目录

～ $ cd /usr/local/Cellar/hadoop/hadoop/ $ mkdir hadoop

安装brew

/usr/local/Cellar/hadoop/ $ /bin/zsh-c“$(curl-fsSL https://gitee.com/cunkai/HomebrewCN/raw/master/Homebrew.sh)” 

建议选择清华大学下载源

按照中文提示进行下去即可，若报错则更改源即可

****此过程花费时间根据你的网络而定****

安装hadoop

Updating Homebrew...==>Downloading https://www.apache.org/dyn/closer.cgi?path=hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz==>Downloadingfromhttp://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz######################################################################## 100.0%?  /usr/local/Cellar/hadoop/3.3.0: 21,686 files, 774.1MB, built in 10 minutes 1 second

*版本默认下载最新，使用暂无无任何影响

配置hadoop文件

雷同于Linux(centos)中的配置

 ~ $ cd /usr/local/Cellar/hadoop/3.3.0/libexec/etc/hadoop

*三个文件均在该目录下

配置  hadoop-env.sh

 /hadoop $ vim hadoop-env.sh :set nu

如图

*第54行去掉# 填上个人的JAVA_HOME的地址即可

*/usr/libexec/java_home 可查JAVA_HOME的信息 复制黏贴即可

配置 core-site.xml

修改core-site.xml 文件参数,配置NameNode的主机名和端口号

<configuration>    <property>        <name>hadoop.tmp.dirname>        <value>/usr/local/Cellar/hadoop/hdfs/tmpvalue>        <description>A base for other temporary directoriesdescription>    property>    <property>        <name>fs.default.namename>        <value>hdfs://localhost:9000value>    property>configuration>

配置 hdfs-site.xm

变量dfs.replication指定了每个HDFS数据库的复制次数。通常为3, 由于我们只有一台主机和一个伪分布式模式的DataNode，将此值修改为1

<configuration> <property> <name>dfs.replicationname> <value>1value> property>configuration>

格式化

只需第一次启动的时候使用

hdfs namenode -format

(此处小编忘记截图)均为INFO或出现successful即可

启动NameNode和DataNode

$ cd /usr/local/Cellar/hadoop/3.1.2/sbin$ ./start-dfs.shStarting namenodes on [localhost]localhost: namenode is running as process 3395.  Stop it first.Starting datanodeslocalhost: datanode is running as process 3494.  Stop it first.Starting secondary namenodes [liuyuanchengdeMacBook-Pro.local]xxxxxMacBook-Pro.local: secondarynamenode is running as process 3628.  Stop it first.2020-11-28 14:54:42,640 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

*小编这里已经开启所以会有所不同

启动yarn服务

./start-yarn.sh

查看服务开启情况

sbin % jps3395 NameNode16901 Jps3494 DataNode6343 NodeManager6248 ResourceManager6587 3628 SecondaryNameNode

*小编这里有个SecondaryNameNode所以有6个服务，常规下是5个服务

启动关闭hadoop服务

./start-all.sh./stop-all.sh

UI界面查看Hadoop环境

http://localhost:9870/

http://localhost:8088/cluster

最后

综上在Mac本地就已经搭建完成hadoop环境，再进行eclipse的开发或运行第一个wordcount均可，相关教程也会在接下来的推文中分享

感谢支持，我们都在路上！！！

*涉及部分网址来自网络*

 ■ 悟以往之不谏 · 知来者之可追

Lop-C