一、hadoop安装
1.安装java
见 https://blog.csdn.net/jiuweiC/article/details/104356751
2. 配置SSH
(1)首先在系统里打开远程登录,位置在 System Preference -> Sharing 中,左边勾选 Remote Login,右边选择 All Users。
如果不执行该步骤 后面会报错
ssh localhost
ssh: connect to host localhost port 22: Connection refused
(2)生成新的keygen否则后面会报错 Permission denied (publickey,password,keyboard-interactive).
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
注册关键字:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
3. 安装hadoop
brew install hadoop
4. 配置
/usr/local/Cellar/hadoop/3.1.0/libexec/etc/hadoop 主要都在这个目录下
a) hadoop-env.sh
输入/usr/libexec/java_home看看你把 Java 装到哪里了:
你会看到类似酱紫结果:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home
/usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home
打开 hadoop-env.sh 文件(位置 etc/hadoop/),找到 # export JAVA_HOME=,改参数如下:
export JAVA_HOME={your java home directory}
把 {your java home directory} 改成你上面查到的 Java 路径,记得去掉注释 #。比如 export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home。
b) core-site.xml
打开 core-site.xml 文件(位置 etc/hadoop/),改参数如下:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
c) hdfs-site.xml
打开 hdfs-site.xml 文件(位置 etc/hadoop/),改参数如下:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
d) mapred-site.xml
打开 mapred-site.xml 文件(位置 etc/hadoop/),改参数如下:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
如果文件后缀是 .xml.example,改为 .xml。
e) yarn-site.xml
打开 yarn-site.xml 文件(位置 etc/hadoop/),改参数如下:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
5.运行
- 1.格式化文件系统:
cd /usr/local/Cellar/hadoop/3.3.0/libexec
bin/hdfs namenode -format
- 2.启动 NameNode 和 DataNode:
sbin/start-dfs.sh
现在你应该可以在浏览器中打开下面的链接看到亲切的 Overview 界面了:
NameNode - http://localhost:9870
下面的命令会报告状态信息,如果无法打开web页面,会提示异常原因。
hdfs dfsadmin -report
2020-08-14 19:30:04,025 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 499963174912 (465.63 GB)
Present Capacity: 402799255552 (375.14 GB)
DFS Remaining: 402799251456 (375.14 GB)
DFS Used: 4096 (4 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (1):
Name: 127.0.0.1:9866 (localhost)
Hostname: 10.43.43.113
Decommission Status : Normal
Configured Capacity: 499963174912 (465.63 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 78992306176 (73.57 GB)
DFS Remaining: 402799251456 (375.14 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.57%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Aug 14 19:30:03 CST 2020
Last Block Report: Fri Aug 14 19:25:39 CST 2020
Num of Blocks: 0
若提示
localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused;
则再次执行格式化命令
bin/hdfs namenode -format
让 HDFS 可以被用来执行 MapReduce jobs:
bin/hdfs dfs -mkdir /user
bin/hdfs dfs -mkdir /user/
把 改成你的用户名,记得去掉 <> 。
- 3.启动 ResourceManager 和 NodeManager:
sbin/start-yarn.sh
现在你应该可以在浏览器中打开下面的链接看到亲切的 All Applications 界面了:
ResourceManager - http://localhost:8088
来自:https://www.jianshu.com/p/0e7f16469d87