说明:
安装JDK版本:jdk1.8.0_231
安装Hadoop版本:3.2.1
1. 安装JDK
https://www.jianshu.com/p/a85658902f26
注意添加export那一行的时候根据自己的jdk版本更改,我的是jdk1.8.0_231.jdk,不是jdk1.8.0_211.jdk。
2. 安装Homebrew和Homebrew-cask
https://juejin.im/post/5cd2a50e518825356d54b847
3. 配置SSH localhost
- 打开系统偏好设置-共享-远程登录(默认情况下关闭,将“远程登录打开即可”);
- 生成密钥对,执行如下命令
ssh-keygen -t rsa
- 执行如下命令,将在当前用户目录中的.ssh文件夹中生成id_rsa文件并且以后可以不使用密码登录:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- 使用下面命令,测试是否可以不适用密码登录:
ssh localhost
4. 安装Hadoop
brew install hadoop
5. 配置Hadoop环境
https://www.cnblogs.com/little-YTMM/p/4394981.html
在目录/usr/local/Cellar/hadoop/3.2.1/libexec/etc/hadoop
下找到hadoop-env.sh
文件中找到其中一行:(有木有盆友不造怎么进入目录啊。。就是打开Finder,然后找最上面工具栏有一个“前往”-->“前往文件夹”-->将路径复制进去即可)
-
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"
将其替换为:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="
- 在目录
/usr/local/Cellar/hadoop/3.2.1/libexec/etc/hadoop
下找到core-site.xml,插入如下代码:
<configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/Cellar/hadoop/hdfs/tmp</value> <description>A base for other temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration>
- 在目录
/usr/local/Cellar/hadoop/3.2.1/libexec/etc/hadoop
下找到mapred-site.xml, 在其中添加:<configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9010</value> </property> </configuration>
- 在目录
/usr/local/Cellar/hadoop/3.2.1/libexec/etc/hadoop
下找到hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
- 在运行后台程序前, 必须格式化新安装的HDFS, 并通过创建存储目录和初始化元数据创新空的文件系统, 执行下面命令:
hdfs namenode -format
然而我【报错】
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
ERROR: JAVA_HOME /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home does not exist.
【分析】
(1)ERROR因为我在文章开头配置JDK时export把目录写成
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home
实际上我的JDK版本是jdk1.8.0_231.jdk
所以通过以下方式打开文件更改目录:
open .bash_profile
然后:
source .bash_profile
echo $JAVA_HOME
(2)WARNING来源于:HADOOP_CONF_DIR路径写的不对
这个要去hadoop-env.sh
文件中找到HADOOP_CONF_DIR,我的文件路径写的是{HADOOP_HOME}/etc/hadoop,检索一下文件路径,看哪里有问题
(base) wangwenzhuodeMacBook-Air:~ wangwenzhuo$ ls ${HADOOP_HOME}/etc/hadoop
ls: /usr/local/Cellar/hadoop/3.2.1/etc/hadoop: No such file or directory
(base) wangwenzhuodeMacBook-Air:~ wangwenzhuo$ ls ${HADOOP_HOME}/etc
ls: /usr/local/Cellar/hadoop/3.2.1/etc: No such file or directory
(base) wangwenzhuodeMacBook-Air:~ wangwenzhuo$ ls $HADOOP_HOME/etc
ls: /usr/local/Cellar/hadoop/3.2.1/etc: No such file or directory
(base) wangwenzhuodeMacBook-Air:~ wangwenzhuo$ ls $HADOOP_HOME
INSTALL_RECEIPT.json NOTICE.txt bin sbin
LICENSE.txt README.txt libexec
(base) wangwenzhuodeMacBook-Air:~ wangwenzhuo$ ls $HADOOP_HOME/libexec
bin hadoop-functions.sh mapred-config.sh yarn-config.sh
etc hdfs-config.sh sbin
hadoop-config.sh libexec share
(base) wangwenzhuodeMacBook-Air:~ wangwenzhuo$ ls $HADOOP_HOME/libexec/etc
hadoop
因此更改为{HADOOP_HOME}/libexec/etc/hadoop
6. 启动后台程序:在/usr/local/Cellar/hadoop/3.2.1/sbin
目录下, 执行如下命令
https://blog.csdn.net/jxq0816/article/details/78736449 参考文章后半部分
start-dfs.sh #启动HDFS
start-yarn.sh
【结果报错】
-bash: start-dfs.sh: command not found
【解决方法】逐步进入目录,在相应文件夹下执行
./start-dfs.sh
./start-yarn.sh
(base) wangwenzhuodeMacBook-Air:hadoop wangwenzhuo$ cd /usr/local/Cellar/hadoop/3.2.1
(base) wangwenzhuodeMacBook-Air:3.2.1 wangwenzhuo$ ls
INSTALL_RECEIPT.json README.txt sbin
LICENSE.txt bin
NOTICE.txt libexec
(base) wangwenzhuodeMacBook-Air:3.2.1 wangwenzhuo$ cd sbin
(base) wangwenzhuodeMacBook-Air:sbin wangwenzhuo$ ls
FederationStateStore refresh-namenodes.sh stop-balancer.sh
distribute-exclude.sh start-all.sh stop-dfs.sh
hadoop-daemon.sh start-balancer.sh stop-secure-dns.sh
hadoop-daemons.sh start-dfs.sh stop-yarn.sh
httpfs.sh start-secure-dns.sh workers.sh
kms.sh start-yarn.sh yarn-daemon.sh
mr-jobhistory-daemon.sh stop-all.sh yarn-daemons.sh
(base) wangwenzhuodeMacBook-Air:sbin wangwenzhuo$ ./start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [wangwenzhuodeMacBook-Air.local]
wangwenzhuodeMacBook-Air.local: Warning: Permanently added 'wangwenzhuodemacbook-air.local,172.27.130.220' (ECDSA) to the list of known hosts.
2019-10-21 10:29:06,840 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
(base) wangwenzhuodeMacBook-Air:sbin wangwenzhuo$ ./start-yarn.sh
Starting resourcemanager
Starting nodemanagers
(base) wangwenzhuodeMacBook-Air:sbin wangwenzhuo$