大数据-Hadoop 3.3.1安装详解

最新推荐文章于 2024-05-05 10:59:28 发布

nihui123

最新推荐文章于 2024-05-05 10:59:28 发布

阅读量1.6k

点赞数 3

分类专栏：云计算与大数据文章标签： hadoop big data hdfs

本文链接：https://blog.csdn.net/nihui123/article/details/122283865

版权

云计算与大数据专栏收录该内容

13 篇文章 1 订阅

订阅专栏

导语
在之前的博客中，讲到了利用Docker安装Hadoop，那么如何使用源码包进行安装呢？这篇博客就来详细说明一下如何利用源码包进行安装。

文章目录

下载包

https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz

解压缩

tar -zxvf hadoop-3.3.1.tar.gz

解压之后的目录如下

在这里插入图片描述

配置文件

cd etc/hadoop // 进入到这目录

在这里插入图片描述

hadoop-env.sh

配置Java的环境变量

mapred-site.xml

添加如下的配置

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9010</value>
    </property>
</configuration>

在这里插入图片描述

hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

在这里插入图片描述

yarn-site.xml(修改警告的阈值默认是95%，这个是非必须参数)

<property>
	<name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
	<value>98.0</value>
</property>

在这里插入图片描述

core-site.xml

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
	<value>/usr/local/Cellar/hadoop/3.3.0/data/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
    </property>
</configuration>

在这里插入图片描述

初始化 namenode

[root@localhost bin]# ./hadoop namenode -format
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

WARNING: /root/hadoop-3.3.1/logs does not exist. Creating.
2022-01-03 09:37:36,145 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.3.1
STARTUP_MSG:   classpath = /root/hadoop-3.3.1/etc/hadoop:/root/hadoop-3.3.1/share/hadoop/common/lib/hadoop-annotations-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/re2j-1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/hadoop-auth-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-collections-3.2.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-security-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jsch-0.1.55.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-text-1.4.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jsr305-3.0.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/woodstox-core-5.3.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/metrics-core-3.2.4.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-api-1.7.30.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/stax2-api-4.2.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/dnsjava-2.1.7.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/curator-framework-4.2.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jersey-json-1.19.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-client-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/failureaccess-1.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerby-config-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-databind-2.10.5.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-xml-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-util-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/netty-3.10.6.Final.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/log4j-1.2.17.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/avro-1.7.7.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-core-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-net-3.6.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/asm-5.0.4.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-codec-1.11.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jakarta.activation-api-1.2.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/json-smart-2.4.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/snappy-java-1.1.8.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/guava-27.0-jre.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-core-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/accessors-smart-2.4.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-server-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-http-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/zookeeper-3.5.6.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerby-util-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-compress-1.19.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/token-provider-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-util-ajax-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/checker-qual-2.5.2.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jettison-1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-servlet-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/paranamer-2.3.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/httpclient-4.5.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-io-2.8.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jersey-core-1.19.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-server-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/commons-lang3-3.7.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-webapp-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/gson-2.2.4.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jersey-servlet-1.19.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/kerb-common-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-annotations-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/curator-client-4.2.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jersey-server-1.19.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jul-to-slf4j-1.7.30.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/httpcore-4.4.13.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/curator-recipes-4.2.0.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-util-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/lib/jetty-io-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/common/hadoop-registry-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/common/hadoop-nfs-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/common/hadoop-kms-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1-tests.jar:/root/hadoop-3.3.1/share/hadoop/hdfs:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/hadoop-annotations-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/re2j-1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/hadoop-auth-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-security-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-text-1.4.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/woodstox-core-5.3.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/netty-all-4.1.61.Final.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/stax2-api-4.2.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/curator-framework-4.2.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-databind-2.10.5.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-xml-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/avro-1.7.7.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-core-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-net-3.6.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/asm-5.0.4.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jakarta.activation-api-1.2.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/json-smart-2.4.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/snappy-java-1.1.8.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/accessors-smart-2.4.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-http-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-compress-1.19.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/okio-1.6.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jettison-1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-servlet-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/paranamer-2.3.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-io-2.8.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-server-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-webapp-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/gson-2.2.4.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-annotations-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/curator-client-4.2.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/curator-recipes-4.2.0.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-util-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/lib/jetty-io-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-client-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.1-tests.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-client-3.3.1-tests.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-3.3.1-tests.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.1-tests.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.1-tests.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn:/root/hadoop-3.3.1/share/hadoop/yarn/lib/websocket-client-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/asm-analysis-9.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/java-util-1.9.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/guice-4.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jetty-client-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/javax.inject-1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jna-5.2.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jersey-client-1.19.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/fst-2.50.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jline-3.9.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/websocket-server-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jetty-jndi-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jetty-plus-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/websocket-common-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/objenesis-2.6.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/snakeyaml-1.26.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jackson-jaxrs-base-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/asm-commons-9.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/websocket-servlet-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/asm-tree-9.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/websocket-api-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.10.5.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/json-io-2.5.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/root/hadoop-3.3.1/share/hadoop/yarn/lib/jetty-annotations-9.4.40.v20210413.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-services-core-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-api-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-router-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-registry-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-client-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-common-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-services-api-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-common-3.3.1.jar:/root/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.1.jar
STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2; compiled by 'ubuntu' on 2021-06-15T05:13Z
STARTUP_MSG:   java = 1.8.0_291
************************************************************/
2022-01-03 09:37:36,161 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2022-01-03 09:37:36,351 INFO namenode.NameNode: createNameNode [-format]
2022-01-03 09:37:37,615 INFO namenode.NameNode: Formatting using clusterid: CID-466e9cc9-254b-4b0f-bbfb-96daefe8c618
2022-01-03 09:37:37,701 INFO namenode.FSEditLog: Edit logging is async:true
2022-01-03 09:37:37,757 INFO namenode.FSNamesystem: KeyProvider: null
2022-01-03 09:37:37,759 INFO namenode.FSNamesystem: fsLock is fair: true
2022-01-03 09:37:37,759 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2022-01-03 09:37:37,773 INFO namenode.FSNamesystem: fsOwner                = root (auth:SIMPLE)
2022-01-03 09:37:37,773 INFO namenode.FSNamesystem: supergroup             = supergroup
2022-01-03 09:37:37,773 INFO namenode.FSNamesystem: isPermissionEnabled    = true
2022-01-03 09:37:37,773 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2022-01-03 09:37:37,773 INFO namenode.FSNamesystem: HA Enabled: false
2022-01-03 09:37:37,845 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-01-03 09:37:37,859 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2022-01-03 09:37:37,859 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2022-01-03 09:37:37,863 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2022-01-03 09:37:37,863 INFO blockmanagement.BlockManager: The block deletion will start around 2022 一月 03 09:37:37
2022-01-03 09:37:37,865 INFO util.GSet: Computing capacity for map BlocksMap
2022-01-03 09:37:37,865 INFO util.GSet: VM type       = 64-bit
2022-01-03 09:37:37,874 INFO util.GSet: 2.0% max memory 1.4 GB = 28.1 MB
2022-01-03 09:37:37,874 INFO util.GSet: capacity      = 2^22 = 4194304 entries
2022-01-03 09:37:37,889 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2022-01-03 09:37:37,889 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2022-01-03 09:37:37,935 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManager: defaultReplication         = 1
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManager: maxReplication             = 512
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManager: minReplication             = 1
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2022-01-03 09:37:37,936 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2022-01-03 09:37:37,937 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2022-01-03 09:37:37,937 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2022-01-03 09:37:37,975 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2022-01-03 09:37:37,976 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2022-01-03 09:37:37,976 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2022-01-03 09:37:37,976 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2022-01-03 09:37:37,994 INFO util.GSet: Computing capacity for map INodeMap
2022-01-03 09:37:37,994 INFO util.GSet: VM type       = 64-bit
2022-01-03 09:37:37,994 INFO util.GSet: 1.0% max memory 1.4 GB = 14.0 MB
2022-01-03 09:37:37,994 INFO util.GSet: capacity      = 2^21 = 2097152 entries
2022-01-03 09:37:37,996 INFO namenode.FSDirectory: ACLs enabled? true
2022-01-03 09:37:37,996 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2022-01-03 09:37:37,996 INFO namenode.FSDirectory: XAttrs enabled? true
2022-01-03 09:37:37,997 INFO namenode.NameNode: Caching file names occurring more than 10 times
2022-01-03 09:37:38,002 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2022-01-03 09:37:38,012 INFO snapshot.SnapshotManager: SkipList is disabled
2022-01-03 09:37:38,017 INFO util.GSet: Computing capacity for map cachedBlocks
2022-01-03 09:37:38,017 INFO util.GSet: VM type       = 64-bit
2022-01-03 09:37:38,018 INFO util.GSet: 0.25% max memory 1.4 GB = 3.5 MB
2022-01-03 09:37:38,018 INFO util.GSet: capacity      = 2^19 = 524288 entries
2022-01-03 09:37:38,030 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2022-01-03 09:37:38,031 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2022-01-03 09:37:38,031 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2022-01-03 09:37:38,043 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2022-01-03 09:37:38,043 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2022-01-03 09:37:38,045 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2022-01-03 09:37:38,045 INFO util.GSet: VM type       = 64-bit
2022-01-03 09:37:38,046 INFO util.GSet: 0.029999999329447746% max memory 1.4 GB = 431.2 KB
2022-01-03 09:37:38,046 INFO util.GSet: capacity      = 2^16 = 65536 entries
2022-01-03 09:37:38,104 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1869460762-127.0.0.1-1641173858092
2022-01-03 09:37:38,131 INFO common.Storage: Storage directory /usr/local/Cellar/hadoop/3.3.0/data/tmp/dfs/name has been successfully formatted.
2022-01-03 09:37:38,200 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/Cellar/hadoop/3.3.0/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2022-01-03 09:37:38,376 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/Cellar/hadoop/3.3.0/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .
2022-01-03 09:37:38,398 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2022-01-03 09:37:38,428 INFO namenode.FSNamesystem: Stopping services started for active state
2022-01-03 09:37:38,429 INFO namenode.FSNamesystem: Stopping services started for standby state
2022-01-03 09:37:38,437 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2022-01-03 09:37:38,437 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/
[root@localhost bin]#

启动

[root@localhost sbin]# ./start-all.sh 
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@localhost sbin]#

解决问题

将start-dfs.sh，stop-dfs.sh两个文件顶部添加以下参数

HDFS_NAMENODE_USER=root

HDFS_DATANODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

YARN_RESOURCEMANAGER_USER=root

YARN_NODEMANAGER_USER=root


 start-yarn.sh，stop-yarn.sh顶部也需添加以下

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn

YARN_NODEMANAGER_USER=root

在这里插入图片描述

$ cd ~/.ssh/        # 若没有该目录，请先执行一次ssh localhost
$ ssh-keygen -t rsa    # 会有提示，都按回车即可
$ cat ./id_rsa.pub >> ./authorized_keys  # 加入授权

修改hadoop-env.sh

添加JAVA_HOME=$JAVA_HOME
在这里插入图片描述

在这里插入图片描述

查看

http://192.168.1.249:8042/node
在这里插入图片描述
http://192.168.1.249:9870/

测试

创建目录和文件

hadoop fs -mkdir -p /input
echo 'hello word' > word.txt 
hadoop fs -put word.txt /input/

使用hadoop 示例：
统计文件中word的数量

hadoop jar /usr/local/Cellar/hadoop/3.3.0/libexec/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.3.0-sources.jar org.apache.hadoop.examples.WordCount

nihui123

关注

3
点赞
踩
6

收藏

觉得还不错? 一键收藏
打赏
1
评论
大数据-Hadoop 3.3.1安装详解

导语 HDFS的核心设计主要包括数据块、数据块复制、数据块的副本存放策略、机架感知、数据块的备份数、安全模式、负载均衡以及心跳机制等功能模块的设计，下面就来看看这些模块都是如何设计的。数据块数据块（Block）是HDFS上存储数据的基本单位。任何一个文件系统其实都是由数据块的概念，只要是文件系统，都会涉及到文件数据的存储问题，当然存储这些文件也需要有相应的数据结构，所以数据块可以看做是文件系统上存储数据的基本单位。在Windows中文件系统有自己的数据块涉及，Linux系统中也有自己的.
复制链接

扫一扫