大数据学习笔记第1课 Hadoop基础理论与集群搭建

最新推荐文章于 2024-04-08 11:07:08 发布
最新推荐文章于 2024-04-08 11:07:08 发布
阅读量327 收藏
点赞数
分类专栏： java 文章标签： hadoop 大数据 学习 算法 spring
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/m0_67392182/article/details/126565628
版权

大数据学习笔记第1课 Hadoop基础理论与集群搭建

最近利用业余时间学习一些大数据的课程，趁周末的时间把学习笔记整理一下，直接上干货吧！

一、环境准备

1、服务器：三台华为云ECS（都是4核8G）

计算机名

内网IP

备注

ecs-ae8a-0001

192.168.0.177

namenode/datanode

ecs-ae8a-0002

192.168.0.56

datanode

ecs-ae8a-0003

192.168.0.59

datanode

2、系统版本：CentOS Linux release 7.5.1804 (Core)
3、修改3台服务器的/etc/hosts
ecs-ae8a-0001

::1	localhost	localhost.localdomain	localhost6	localhost6.localdomain6
127.0.0.1	localhost	localhost.localdomain	localhost4	localhost4.localdomain4
#127.0.0.1	ecs-ae8a-0001	ecs-ae8a-0001

192.168.0.177	hadoop01	ecs-ae8a-0001
192.168.0.56	hadoop02	ecs-ae8a-0002
192.168.0.59	hadoop03	ecs-ae8a-0003

ecs-ae8a-0002

::1	localhost	localhost.localdomain	localhost6	localhost6.localdomain6
127.0.0.1	localhost	localhost.localdomain	localhost4	localhost4.localdomain4
#127.0.0.1	ecs-ae8a-0002	ecs-ae8a-0002

192.168.0.177	hadoop01	ecs-ae8a-0001
192.168.0.56	hadoop02	ecs-ae8a-0002
192.168.0.59	hadoop03	ecs-ae8a-0003

ecs-ae8a-0003

::1	localhost	localhost.localdomain	localhost6	localhost6.localdomain6
127.0.0.1	localhost	localhost.localdomain	localhost4	localhost4.localdomain4
#127.0.0.1	ecs-ae8a-0003	ecs-ae8a-0003

192.168.0.177	hadoop01	ecs-ae8a-0001
192.168.0.56	hadoop02	ecs-ae8a-0002
192.168.0.59	hadoop03	ecs-ae8a-0003

注意：
1、修改完毕确保hadoop01、hadoop02、hadoop03都能互相ping通。
2、需要把以上3图中的以下内容前通过#注释掉，不然启动集群时会导致所有datanode的clusterID值相同，会导致在hadoop图形化界面上只有一个datanode。

ecs-ae8a-0001

#127.0.0.1	ecs-ae8a-0001	ecs-ae8a-0001

ecs-ae8a-0002

#127.0.0.1	ecs-ae8a-0002	ecs-ae8a-0002

ecs-ae8a-0003

#127.0.0.1	ecs-ae8a-0003	ecs-ae8a-0003

二、下载JDK

1、本地浏览器进入甲骨文官网：https://www.oracle.cn
2、创建一个Oracle帐户，然后登录，如果已有用户，则直接登录。
点击页面顶端的[查看账户]，出现下图界面。
在这里插入图片描述
3、点击登录按钮，会进入登录界面如下图：
在这里插入图片描述
4、输入你的用户名和密码进行登录，登录成功后，点击页面顶端的[产品]菜单项，出现子级菜单项，如下图：
在这里插入图片描述
5、点击上图中的Java选项，进入Java介绍页面，如下图：
在这里插入图片描述
6、点击上图中顶端右侧的[下载Java]按钮，进入Java下载页面，如下图：
在这里插入图片描述
7、从上图中可以看出最新版为JavaSE15，可以点击上图中的[JDK Download]链接，出现下图：
在这里插入图片描述
8、如上图根据你的目标操作系统选择相应的版本，因为我们是在Linux下安装，可以点击[jdk-15.0.1_linux-x64_bin.rpm]，首次下载会弹出一个对话框，如下图：
在这里插入图片描述
9、如上图，首先勾选[I reviewed and accept the Oracle Technology Network License Agreement for Oracle Java SE]选项，直接点击[Download jdk-15.0.1_linux-x64_bin.rpm]按钮出现下载对话框，如下图：
在这里插入图片描述
如果想下载到本地可以点击[下载]。因为我这里是要在centos系统中直接下载，因此只要复制下载地址就行。
10、回到xshell的终端系统中，新建一个/opt/soft目录用于存放下载的软件。
11、 通过wget下载jdk的rpm包，如下：

wget https://download.oracle.com/otn-pub/java/jdk/15.0.1+9/51f4f36ad4ef43e39d0dfdbaf6549e32/jdk-15.0.1_linux-x64_bin.rpm?AuthParam=1607139610_2190d5cdaadb068f38b24911e869ae6b

下载过程如下图：
在这里插入图片描述
下载后的文件名会带AuthParm后缀，需要先重命名一下，如下：

mv jdk-15.0.1_linux-x64_bin.rpm?AuthParam=1607139610_2190d5cdaadb068f38b24911e869ae6b jdk-15.0.1_linux-x64_bin.rpm

12、3台服务器执行相同的下载操作，当然也可以使用scp命令把rpm包直接远程复制到其他2台服务器上，如下：

scp jdk-15.0.1_linux-x64_bin.rpm root@192.168.0.59:/opt/soft/

需要输入用户密码进行验证，如下图：
在这里插入图片描述
这样在192.168.0.59服务器上就有jdk的rpm包了，如下图：
在这里插入图片描述

三、安装JDK

1、在xshell终端中执行rpm命令安装rpm包，如下：

rpm -ivh jdk-15.0.1_linux-x64_bin.rpm

安装过程如下图：
在这里插入图片描述
2、通过java -version验证jdk是否安装成功，如下图：
在这里插入图片描述
3、3台服务器执行相同的安装和验证操作。

四、下载hadoop

1、先进入apache官网，地址为：http://www.apache.org/，如下图:
在这里插入图片描述
2、点击首页的Projects菜单项会弹出子菜单列表，如下图：
在这里插入图片描述
3、选中Project List菜单项，进入项目列表页面，如下图：
在这里插入图片描述
4、在项目列表页面中找H字母开头的，目前第一项就是Hadoop，点击进入Hadoop主页，如下图：
在这里插入图片描述
5、 点击Hadoop主页上的Download按钮，进入Hadoop下载页面，如下图：
在这里插入图片描述
6、在Hadoop的下载页面中选择最新版的二进制下载，这里是3.3.0版本的binary，会进入hadoop-3.3.0.tar.gz的镜像下载地址页面，如下图：
在这里插入图片描述
7、复制其中的一个下载地址，然后回到xshell的终端系统中，使用wget进行下载，为了下载速度我在华为云服务器上使用的是另一个镜像地址，如下：

wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz

下载过程如下图：
在这里插入图片描述
8、使用tar -xzvf命令解压缩hadoop-3.3.0.tar.gz，如下：

tar -xzvf haddop-3.3.0.tar.gz

解压后如下图：
在这里插入图片描述
9、 三台服务器执行相同的下载与解压操作。
10、新建一个程序目录/program，如下：

mkdir /program

11、 把解压的hadoop-3.3.0目录复制到/program/目录下，如下：

cp -r hadoop-3.3.0 /program/

12、3台服务器执行相同的创建目录与复制操作。

五、安装hadoop集群

1、切换当前目录为/program/hadoop-3.3.0，了解一下目录内容。
2、切换当前目录为/program/hadoop-3.3.0/bin，理解一下目录下的内容。
3、通过./hadoop命令执行hadoop安装。这是会出现未找到JAVA_HOME错误信息：

ERROR: JAVA_HOME is not set and could not be found.

4、因为之前已经安装了jdk，只是没有配置JAVA_HOME环境变量而已，这里先通过which java找到jdk的安装目录。which java执行后输出的目录有可能是软链接，因此需要通过ll -a 文件目录，看看有没有->符号，如果有则说明是软链接。这里找到的JAVA_HOME为/usr/java/jdk-15.0.1。如下图：
在这里插入图片描述
5、为了解决这个文件首先切换目录至/program/hadoop-3.3.0/etc/hadoop目录
6、然后通过vim命令编辑文件/program/hadoop-3.3.0/etc/hadoop/hadoop-env.sh。找到

# export JAVA_HOME=

取消前面的#，改为

export JAVA_HOME=/usr/java/jdk-15.0.1

保存hadoop-env.sh，并退出。
7、切换回/program/hadoop-3.3.0/bin目录，使用./hadoop version命令查看hadoop的版本，如下图：
在这里插入图片描述
8、3台服务器执行相同的配置JAVA_HOME操作。

六、打通3台服务器的免密登录

1、目标是在namenode(hadoop01)服务器上能够免密登录2个datanode(hadoop02、hadoop03)服务器。
2、在namenode(hadoop01)服务器上通过ssh-keygen -t rsa命令生成密钥对，如下：

ssh-keygen -t rsa

执行过程如下图：
在这里插入图片描述
这样在~/.ssh/目录下就会有几个文件
id_rsa是私钥文件
id_rsa.pub是公钥文件
3、通过ssh-copy-id命令把公钥内容发送至2个datanode服务器，这是还是会再让输入一次密码的。如下：
在这里插入图片描述
192.168.0.56对应的是hadoop02，为了确保能够在hadoop01上直接ssh登录hadoop02，需要在hadoop01上再执行一次
ssh-copy-id hadoop02
4、通过ssh命令验证是否能够直接登录，如下图：
在这里插入图片描述
5、 重复上面的操作实现能在hadoop01上直接免密登录hadoop03。

七、hadoop集群配置

1、namenode(hadoop01、hadoop02、hadoop03)的配置，设置namenode的IP地址。
通过vim命令打开/program/hadoop-3.3.0/etc/hadoop/core-site.xml文件，在中增加如下内容：

<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://192.168.0.177:9000</value>
	</property>
</configuration>

注意：为了实现3台服务器都要设置，可以现在hadoop01上修改后，再把core-site.xml文件复制到另外2台服务器上。

2、datanode(hadoop01、hadoop02、hadoop03)的配置，设置数据存储目录。
通过vim命令打开/program/hadoop-3.3.0/etc/haddop/hdfs-site.xml文件，在中增加的内容如下：

<configuration>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>/home/data</value>
	</property>
</configuration>

注意：可以现在hadoop01服务器上修改完毕，再把hdfs-site.xml复制到另外2台服务器。

八、hadoop集群初始化

首先进入hadoop01并切换目录至/program/hadoop-3.3.0/bin/
执行以下命令进行集群初始化，其中-format后的名字可以自己定义，这里定义为csdn

./hdfs namenode -format csdn

执行日志如下：

2020-12-06 22:04:04,536 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ecs-ae8a-0001/192.168.0.177
STARTUP_MSG:   args = [-format, csdn]
STARTUP_MSG:   version = 3.3.0
STARTUP_MSG:   classpath = /program/hadoop-3.3.0/etc/hadoop:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:
/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsr305-3.0.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-compress-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-servlet-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-databind-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-client-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/stax2-api-3.1.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/gson-2.2.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-server-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/httpcore-4.4.10.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-core-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-lang3-3.7.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/log4j-1.2.17.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/nimbus-jose-jwt-7.9.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/netty-3.10.6.Final.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-servlet-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-net-3.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/dnsjava-2.1.7.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-json-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/accessors-smart-1.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/failureaccess-1.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-config-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/re2j-1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-text-1.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-annotations-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-core-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jettison-1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsch-0.1.55.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/token-provider-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/metrics-core-3.2.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-codec-1.11.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-webapp-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/zookeeper-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/paranamer-2.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/curator-framework-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsp-api-2.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-cli-1.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/hadoop-auth-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-xml-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-io-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/checker-qual-2.5.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/javax.activation-api-1.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/httpclient-4.5.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/asm-5.0.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/json-smart-2.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/curator-recipes-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-security-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-io-2.5.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-server-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-core-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-util-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-server-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-http-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/curator-client-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-common-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/avro-1.7.7.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/hadoop-annotations-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-kms-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-registry-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-nfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-compress-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-servlet-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-databind-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/gson-2.2.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/httpcore-4.4.10.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/nimbus-jose-jwt-7.9.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-net-3.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/re2j-1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-text-1.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-annotations-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-core-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jettison-1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-webapp-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/paranamer-2.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-framework-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-auth-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-xml-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-io-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/javax.activation-api-1.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/httpclient-4.5.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/asm-5.0.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/json-smart-2.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-recipes-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-security-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-io-2.5.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/netty-all-4.1.50.Final.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-util-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-server-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-http-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-client-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/okio-1.6.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/avro-1.7.7.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-annotations-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-client-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-client-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-client-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/asm-analysis-7.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-api-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/asm-commons-7.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax.inject-1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-client-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/json-io-2.5.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-jaxrs-base-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-servlet-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jna-5.2.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-annotations-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jline-3.9.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/java-util-1.9.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-jndi-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/objenesis-2.6.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jakarta.activation-api-1.2.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-common-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/asm-tree-7.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jersey-client-1.19.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-server-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-plus-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/guice-4.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/fst-2.50.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-client-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-api-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-services-core-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-common-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-registry-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-services-api-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-router-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-common-3.3.0.jarSTARTUP_MSG:   build = https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af; compiled by 'brahma' on 2020-07-06T18:44Z
STARTUP_MSG:   java = 15.0.1
************************************************************/
2020-12-06 22:04:04,544 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2020-12-06 22:04:04,613 INFO namenode.NameNode: createNameNode [-format, csdn]
2020-12-06 22:04:04,949 INFO namenode.NameNode: Formatting using clusterid: CID-6fd94f42-31a1-4fb3-933b-de6e91ba3bfb
2020-12-06 22:04:04,967 INFO namenode.FSEditLog: Edit logging is async:true
2020-12-06 22:04:04,989 INFO namenode.FSNamesystem: KeyProvider: null
2020-12-06 22:04:04,990 INFO namenode.FSNamesystem: fsLock is fair: true
2020-12-06 22:04:04,990 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: fsOwner                = root (auth:SIMPLE)
2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: supergroup             = supergroup
2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: isPermissionEnabled    = true
2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: HA Enabled: false
2020-12-06 22:04:05,048 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2020-12-06 22:04:05,055 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2020-12-06 22:04:05,055 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2020-12-06 22:04:05,057 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2020-12-06 22:04:05,057 INFO blockmanagement.BlockManager: The block deletion will start around 2020 Dec 06 22:04:05
2020-12-06 22:04:05,058 INFO util.GSet: Computing capacity for map BlocksMap
2020-12-06 22:04:05,058 INFO util.GSet: VM type       = 64-bit
2020-12-06 22:04:05,059 INFO util.GSet: 2.0% max memory 1.9 GB = 39.1 MB
2020-12-06 22:04:05,059 INFO util.GSet: capacity      = 2^22 = 4194304 entries
2020-12-06 22:04:05,069 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2020-12-06 22:04:05,069 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2020-12-06 22:04:05,073 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2020-12-06 22:04:05,073 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2020-12-06 22:04:05,073 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2020-12-06 22:04:05,073 INFO blockmanagement.BlockManager: defaultReplication         = 3
2020-12-06 22:04:05,073 INFO blockmanagement.BlockManager: maxReplication             = 512
2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: minReplication             = 1
2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2020-12-06 22:04:05,089 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2020-12-06 22:04:05,089 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2020-12-06 22:04:05,089 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2020-12-06 22:04:05,090 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2020-12-06 22:04:05,098 INFO util.GSet: Computing capacity for map INodeMap
2020-12-06 22:04:05,098 INFO util.GSet: VM type       = 64-bit
2020-12-06 22:04:05,098 INFO util.GSet: 1.0% max memory 1.9 GB = 19.6 MB
2020-12-06 22:04:05,098 INFO util.GSet: capacity      = 2^21 = 2097152 entries
2020-12-06 22:04:05,101 INFO namenode.FSDirectory: ACLs enabled? true
2020-12-06 22:04:05,101 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2020-12-06 22:04:05,101 INFO namenode.FSDirectory: XAttrs enabled? true
2020-12-06 22:04:05,101 INFO namenode.NameNode: Caching file names occurring more than 10 times
2020-12-06 22:04:05,104 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2020-12-06 22:04:05,106 INFO snapshot.SnapshotManager: SkipList is disabled
2020-12-06 22:04:05,108 INFO util.GSet: Computing capacity for map cachedBlocks
2020-12-06 22:04:05,108 INFO util.GSet: VM type       = 64-bit
2020-12-06 22:04:05,109 INFO util.GSet: 0.25% max memory 1.9 GB = 4.9 MB
2020-12-06 22:04:05,109 INFO util.GSet: capacity      = 2^19 = 524288 entries
2020-12-06 22:04:05,115 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2020-12-06 22:04:05,115 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2020-12-06 22:04:05,115 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2020-12-06 22:04:05,118 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2020-12-06 22:04:05,118 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2020-12-06 22:04:05,119 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2020-12-06 22:04:05,119 INFO util.GSet: VM type       = 64-bit
2020-12-06 22:04:05,119 INFO util.GSet: 0.029999999329447746% max memory 1.9 GB = 600.9 KB
2020-12-06 22:04:05,119 INFO util.GSet: capacity      = 2^16 = 65536 entries
2020-12-06 22:04:05,135 INFO namenode.FSImage: Allocated new BlockPoolId: BP-549884843-192.168.0.177-1607263445130
2020-12-06 22:04:05,149 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
2020-12-06 22:04:05,167 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2020-12-06 22:04:05,240 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .
2020-12-06 22:04:05,251 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2020-12-06 22:04:05,255 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2020-12-06 22:04:05,255 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ecs-ae8a-0001/192.168.0.177
************************************************************/

九、启动hadoop集群

1、为了集中启动，可以修改hadoop01下的/program/hadoop-3.3.0/etc/hadoop/workers文件，具体修改为：

hadoop01
hadoop02
hadoop03

注意：确保已在各自的/etc/hosts文件中进行了ip映射，确保能够互相ping通。
切换目录至/program/hadoop-3.3.0/sbin/
2、执行命令./start-dfs.sh，这时会报错误，如下图：
在这里插入图片描述
4、为了解决上面的错误，需要修改/program/hadoop-3.3.0/etc/hadoop/hadoop-env.sh文件内容。
在export JAVA_HOME=/usr/java/jdk-15.0.1之后增加一下3行内容

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

然后保存退出。
5、重新执行命令./start-dfs.sh，如下图：
在这里插入图片描述
6、通过jps命令查看守护进程是否成功启动，三台服务器的输出如下图：
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
7、在浏览器中使用hadoop图形界面查看，在（另一台windows的华为云ECS中，本地电脑需要使用公网IP）浏览器地址在中输入http://192.168.0.177:9870/，出现下图：
在这里插入图片描述
8、选择顶部菜单的Datanodes，出现下图：
在这里插入图片描述

十、hadoop集群第一坑填坑记。

由于第一次操作不熟练，因此搭建的过程中不可避免出现一些问题。关键的一个问题就是一切启动都正常，但是在hadoop的图片管理界面中只看到1个数据节点。通过分析发现是各个数据节点的clusterID值相同导致（/home/data/current/VERSION文件中记录）。中间也进行过多次格式化包括网上说的删除3台服务器/tmp/目录内容和/home/data/目录的内容，再重新初始化还是不行。最后通过修改/etc/hosts文件再格式化就可以了。

需要把3台服务器/etc/hosts的以下内容前通过#注释掉，不然启动集群时会导致所有datanode的clusterID值相同，会导致在hadoop图形化界面上只有一个datanode。

ecs-ae8a-0001

#127.0.0.1	ecs-ae8a-0001	ecs-ae8a-0001

ecs-ae8a-0002

#127.0.0.1	ecs-ae8a-0002	ecs-ae8a-0002

ecs-ae8a-0003

#127.0.0.1	ecs-ae8a-0003	ecs-ae8a-0003

结束

至此一个简单的hadoop集群搭建完毕。希望对初学的朋友能有个参考。最后感谢一下csdn大数据的老师吧，毕竟有个人带着学要快的多。如果觉得有帮助点个赞吧~

bp粉
关注
专栏目录
大数据笔记（三）：HDFS集群搭建-伪分布式模式
Lansonli（蓝深李）的博客
04-24 3799
HDFS集群搭建-伪分布式模式
01Hadoop学习笔记-Hadoop集群搭建
qq_40850266的博客
03-07 1727
Hadoop是apache的一个大数据存储计算的开源软件
Hadoop原理与集群搭建笔记
a814046606的博客
09-25 507
一、理论部分知识 1、HDFS ：Hadoop DIstributed File System简写。 易于扩展的分布式文件系统 运行在大量普通廉价机器上（成本低） ，提供容错的机制（可靠性高） 2、HDFS优点： 高容错性 适合大数据批处理（移动计算不移动数据、数据位置暴露给计算框架、存储量大、百万规模以上的文件数量、10k节点规模） 流式文件访问（一次写入，多次读取，保证数据一致性） 构建成本低...
hadoop怎么学习？hadoop学习计划
weixin_34388207的博客
03-13 265
Hadoop是在分布式服务器集群上存储海量数据并运行分布式分析应用的一个平台，其核心部件是HDFS与MapReduce。HDFS是一个分布式文件系统，可对数据系统进行分布式储存读取；MapReduce是一个计算框架，通过对计算任务的拆分，再根据任务调度器，对任务进行分布式计算。Hadoop是大数据开发必不可少的框架技术，因此，想要学好大数据，必须要掌握Hadoop相关知识，那...
大数据---Hadoop集群搭建学习（干货）
yidianyidei的博客
07-04 843
文章目录Hadoop 集群搭建学习1 Hadoop发展史2 Hadoop三大发行版本3 Hadoop的优势（4高）4 Hadoop组成（面试重点）5 HDFS架构概述6 MapReduce架构概述7 大数据技术生态体系8 推荐系统框架图一 装hadoop&Java1创建文件夹2 安装JDK1.卸载现有JDK2.查看软件包3.解压安装包4 配置环境变量Grep案例WordCount案例二 伪分布式运行模式启动HDFS并运行MapReduce程序配置集群*启动集群启动YARN并运行MapRedu
hadoop搭建(二)——集群搭建
qq_22343953的博客
04-20 484
文章目录hadoop搭建(二)hadoop集群搭建目的预先准备安装配置无安全模式的hadoop配置hadoop守护进程环境配置hadoop守护进程管理节点情况运行hadoop集群hadoop运行hadoop停止 hadoop搭建(二) 本文档翻译hadoop的集群搭建，说明来自：Hadoop Cluster Setup hadoop集群搭建 目的 这篇文档说明了从一些节点到数以千记的大量节点的安装...
大数据学习笔记（三）——大数据处理框架Hadoop
stepondust 的课堂
01-25 3847
第三章 大数据处理框架Hadoop 3.1 概述 3.1.1 Hadoop简介 Hadoop是Apache软件基金会旗下的一个开源分布式计算平台，为用户提供了系统底层细节透明的分布式基础架构。Hadoop 是基于Java语言开发的，具有很好的跨平台特性，并且可以部署在廉价的计算机集群中。 Hadoop 的核心是分布式文件系统HDFS（Hadoop Distributed File System）和MapReduce……
笔记：分布式大数据技术原理（一）Hadoop 框架
WeeeicheN 的博客
07-20 1255
前言-从 Hadoop 说起 什么是 Hadoop Apache Hadoop 软件库是一个框架，它允许使用简单的编程模型，实现跨计算机集群的大型数据集的分布式处理。它最初的设计目的是为了检测和处理应用程序层的故障，从单个机器扩展到数千台机器（这些机器可以是廉价的），每个机器提供本地计算和存储，而不是依靠硬件提供高可用性。 Hadoop 中有3个核心组件： 分布式文件系统：HDFS —— 实现将文件分布式存储在很多的服务器上 分布式运算编程框架：MapReduce —— 实现在很多机器上...
Hadoop学习笔记
weixin_45906989的博客
08-13 1566
Handoop学习笔记
传智黑马赵星老师hadoop七天课程资料笔记-第一天(全)
06-12
【标题】"传智黑马赵星老师hadoop七天课程资料笔记-第一天(全)" 提供的是一份关于Hadoop的全面学习资料，主要聚焦在Hadoop的第一天课程内容。这个资源包涵盖了从基础概念到实际操作的多个方面，旨在帮助初学者快速...
hadoop问题：对‘xdrmem_create’、‘xdr_float’未定义的引用
SundayO的博客
11-14 1878
[WARNING] /usr/bin/c++ -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -rdynamic CMakeFiles/wordcount-nopipe.dir/main/native/examples/impl/wordcount-nopipe.cc.o -o examples/wordcount-...
hadoop集群搭建学习总结（一）
qq_39063243的博客
09-20 5300
Hadoop是Apache旗下的一个用java语言实现开源软件框架，是一个开发和运行处理大规模数据的软件平台。允许使用简单的编程模型在大量计算机集群上对大型数据集进行分布式处理。狭义上说，Hadoop指Apache这款开源框架，它的核心组件有：HDFS（分布式文件系统）：解决海量数据存储YARN（作业调度和集群资源管理的框架）：解决资源任务调度MAPREDUCE（分布式运算编程框架）：解决海量数据计算hadoop集群的搭建即为在集群中安装以上的3个组件。
大数据Hadoop之——Hadoop图形化管理系统Hue（Hue环境部署）
匠人精神，持之以恒！
04-19 7044
文章目录 一、概述 Hue是一个开源的Apache Hadoop UI系统，最早是由Cloudera Desktop演化而来，由Cloudera贡献给开源社区，它是基于Python Web框架Django实现的。通过使用Hue我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据，例如操作HDFS上的数据，运行MapReduce Job等等。 Hue官方网站：https://gethue.com/ 官方安装文档：https://docs.gethue.com/administrato
Hadoop图形管理系统（一）
最新发布
weixin_58647590的博客
04-08 809
Hadoop图形管理系统（一） JDK和Maven安装
hadoop namenode -format 格式化报错
chichichichi的博客
04-11 2808
18/04/11 21:42:07 WARN namenode.FSEditLog: No class configured for master1.hadoop, dfs.namenode.edits.journal-plugin.master1.hadoop is empty18/04/11 21:42:07 ERROR namenode.NameNode: Failed to start...
hadoop namenode格式化失败
weixin_42496757的博客
02-16 4507
执行hadoop namenode -format格式化命令后,报错: Call From hadoop01/192.168.195.134 to hadoop02:8485 failed on connection exception: java.net.Connectxception: 拒绝连接; 详细日志: 19/02/16 12:29:51 INFO namenode.NameNo...
Hadoop-2.4.1学习之NameNode -format源码分析
skyWalker_ONLY
09-16 6133
Hadoop-2.4.1 namenode格式化的源代码分析
NameNode格式化失败问题的解决
赶路人儿
11-05 3667
NameNode格式化失败，查看日志，报如下错： 15/04/08 10:05:43 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]15/04/08 10:05:43 INFO namenode.NameNode: createNameNode [-format]15/04/08 10:05
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值