hadoop搭建全分布环境踩过的坑

环境

平台：virtual Box
系统：Ubuntu20.04
jdk：jdk1.8.0_171
Hadoop：hadoop-3.0.0

问题及解决方案

1.创建的三台虚拟机之间ping不通

虚拟机采用桥接模式

2.运行start-dfs.sh时，出现rcmd:socket:Permission denied错误

在/etc/pdsh/下新建rcmd_default文件，内容为ssh

3.运行start-all.sh时，启动namenode和datanode出现ssh exited with exit code 1

3.1 每一个公钥都要写入authorized_keys文件

	ssh-keygen
	ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu01
	ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu02
	ssh ubuntu02
	cat id_rsa.pub >> authorized_keys
	scp authorized_keys ubuntu03:~/.ssh/
	cat id_rsa.pub >> authorized_keys
	scp authorized_keys ubuntu01:~/.ssh/
	scp authorized_keys ubuntu02:~/.ssh/

3.2 环境变量配全

	#java
	export JAVA_HOME=/usr/local/java/jdk1.8.0_171
	export JRE_HOME=$JAVA_HOME/jre
	export JAVA_LIBRARY_PATH=/usr/local/java/jdk1.8.0_171/lib/native
	export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
	export PATH=$JAVA_HOME/bin:$PATH

	#hadoop
	export HADOOP_HOME=/usr/app/hadoop-3.0.0
	export PATH=$HADOOP_HOME/bin:$PATH
	export HADOOP_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
	export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
	export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
	export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"

4.namenode无法启动

namenode多次格式化后，会无法启动
解决方法：删除data下的current文件，重新进行格式化

5.将hadoop上传至其他节点出现permission denied

将上传的文件和上传至目录权限改为777，重启

6.格式化namenode出现data文件夹不可用的情况

将/home/hadoop/namenode/下的data文件夹权限改为777