hadoop本地模式部署_hadoop本地模式安装及配置

创建普通用户

#仅测试环境使用

hostnamectl set-hostname centos-hadoop01

chmod 777 /opt/

chmod 777 /etc/profile

#仅测试环境使用

sudo su -

useradd centos

echo "centos:centos" | chpasswd

su - centos

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cd ~/.ssh

cat id_rsa.pub >> authorized_keys

chmod 755 authorized_keys

ssh localhost

下载jdk

可复制标红处链接下载

wget https://download.oracle.com/otn/java/jdk/8u201-b09/42970487e3af4f5aa5bca3f542482c60/jdk-8u201-linux-x64.tar.gz?AuthParam=1575547827_c9c9f7cea2ec927a68265154d5dd9325

正常解压即可

tar xf jdk-8u201-linux-x64.tar.gz

echo -e '\n#配置java环境变量' >> /etc/profile

echo -e 'export JAVA_HOME=/opt/jdk1.8.0_201' >> /etc/profile

echo -e 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile

source /etc/profile

java -version

下载hadoop

wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz

tar xf hadoop-3.2.1.tar.gz

安装配置hadoop

core-site.xml

vim /opt/hadoop-3.2.1/etc/hadoop/core-site.xml

fs.defaultFS

hdfs://s101:9000

hadoop.tmp.dir

/opt/module/hadoop/data/tmp

添加变量

echo -e '\n#配置hadoop环境变量' >> /etc/profile

echo -e 'export HADOOP_HOME=/opt/hadoop-3.2.1' >> /etc/profile

echo -e 'export PATH=$PATH:$HADOOP_HOME/bin' >> /etc/profile

echo -e 'export PATH=$PATH:$HADOOP_HOME/sbin' >> /etc/profile

source /etc/profile

core-site.xml

vim /opt/hadoop/etc/hadoop/core-site.xml

fs.defaultFS

hdfs://s101:9000

hadoop.tmp.dir

/opt/module/hadoop/data/tmp

hdfs-site.xml

vim /opt/hadoop-3.2.1/etc/hadoop/hdfs-site.xml

dfs.replication

1

hadoop-env.sh

vim /opt/hadoop-3.2.1/etc/hadoop/hadoop-env.sh

echo 'export JAVA_HOME=/opt/jdk1.8.0_201' >> /opt/hadoop-3.2.1/etc/hadoop/hadoop-env.sh

mapred-site.xml

vim /opt/hadoop-3.2.1/etc/hadoop/mapred-site.xml

mapreduce.framework.name

yarn

yarn-site.xml

vim /opt/hadoop-3.2.1/etc/hadoop/yarn-site.xml

yarn.resourcemanager.hostname

s101

yarn.nodemanager.aux-services

mapreduce_shuffle

hadoop逻辑格式化

hdfs namenode -format

测试hadoop

启动namenode:

start-all.sh

检查里程

jps

浏览器打开检查

http://ip:50070/

错误

Starting namenodes on [s101]

ERROR: Attempting to operate on hdfs namenode as root

ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.

Starting datanodes

ERROR: Attempting to operate on hdfs datanode as root

ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.

Starting secondary namenodes [s101]

ERROR: Attempting to operate on hdfs secondarynamenode as root

ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

出现以上问题，表明不接受root用户启动hdfs。把sbin/start-dfs.sh和sbin/stop-dfs.sh在文件头部追加以下内容

HDFS_DATANODE_USER=root

HADOOP_SECURE_DN_USER=hdfs

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root