hadoop伪分布模式

最新推荐文章于 2023-04-30 11:17:13 发布
最新推荐文章于 2023-04-30 11:17:13 发布
阅读量460 收藏 1
点赞数
分类专栏： 大数据 文章标签： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_45925324/article/details/111386401
版权

3种模式下组件配置的区别

组件名称属性名称单机模式伪分布模式集群模式
Commonfs.defaultFSfile:///(默认)localhost:9000master:9000
HDFSdfs.replicationN/A12
MapReducemapreduce.framework.namelocal(默认)yarnyarn
yarn

yarn.resourcemanager.hostname

yarn.nodemanager.aux_service

N/A

N/A

localhost

mapreduce_shuffle

localhost

mapreduce_shuffle

1.安装jdk

#找到jdk安装包(jdk-8u261-linux-x64.tar.gz)解压
#/usr/local/自己想要解压的路径

tar - zxvf jdk-8u261-linux-x64.tar.gz -C /usr/local/

#进入已解压的路径

cd /usr/local/
ls

#可以查看到jdk1.8.0_261(解压成功)
#配置jdk的环境变量

vim ~/.bashrc

#添加以下两行内容
#其中JAVA_HOME后面的内容是自己jdk的安装路径

export JAVA_HOME=/usr/local/jdk1.8.0_261
export PATH=$PATH:$JAVA_HOME/bin

#使环境变量生效

source ~/.bashrc

 

安装ssh

root@student-virtual-machine:~# apt-get install openssh-server
root@student-virtual-machine:~# ssh-keygen -t rsa
root@student-virtual-machine:~/.ssh# cat id_rsa.pub >> authorized_keys
root@student-virtual-machine:~/.ssh# ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:6D7w/Y6900m3laVWcsaPMDrdhVpyEuqRUrhj7Fum8lE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-112-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage


148 packages can be updated.
122 updates are security updates.

New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@student-virtual-machine:~#

 

3安装hadoop伪分布模式

#解压hadoop安装包

root@student-virtual-machine:/mnt/hgfs/share# tar -zxvf hadoop-2.7.6.tar.gz  -C /usr/local/

#进入解压后的位置

root@student-virtual-machine:/mnt/hgfs/share# cd /usr/local
root@student-virtual-machine:/usr/local# ls
bin  etc  games  hadoop-2.7.6  include  jdk1.8.0_261  lib  man  sbin  share  src

#配置环境变量
root@student-virtual-machine:/usr/local# vim ~/.bashrc
#添加下面两行内容

export HADOOP_HOME=/usr/local/hadoop-2.7.6
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin


root@student-virtual-machine:/usr/local#source ~/.bashrc
root@student-virtual-machine:/usr/local/hadoop-2.7.6# cd etc
root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc# ls
hadoop
root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc# cd hadoop
root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# ls
capacity-scheduler.xml  hadoop-env.sh               httpfs-env.sh            kms-env.sh            mapred-env.sh               ssl-server.xml.example
configuration.xsl       hadoop-metrics2.properties  httpfs-log4j.properties  kms-log4j.properties  mapred-queues.xml.template  yarn-env.cmd
container-executor.cfg  hadoop-metrics.properties   httpfs-signature.secret  kms-site.xml          mapred-site.xml.template    yarn-env.sh
core-site.xml           hadoop-policy.xml           httpfs-site.xml          log4j.properties      slaves                      yarn-site.xml
hadoop-env.cmd          hdfs-site.xml               kms-acls.xml             mapred-env.cmd        ssl-client.xml.example
root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# 
root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# vim hadoop-env.sh
#在里面找到export JAVA_HOME=将自己的jdk路径写入

export JAVA_HOME=/usr/local/jdk1.8.0_261

注意：core-site.xml中的

<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.7.6/tmp</value>这个tmp是我们自己创建的目录，需要自己创建

root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# vim core-site.xml 

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.7.6/tmp</value>
</property>
</configuration>

注意：hdf-site.xml中的

<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop-2.7.6/tmp/name</value>这个name也是我们自己创建的目录
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop-2.7.6/tmp/data</value>这个data也是我们自己创建的目录

root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# vim hdfs-site.xml 

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop-2.7.6/tmp/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop-2.7.6/tmp/data</value>
</property>
</configuration>

root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# vim mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

在yarn-site.xml中

<name>yarn.resourcemanager.hostname</name>
<value>192.168.100.136</value>
原本是
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
但是会出现使用http://localhost:8088打不开web控制台，所以我们改成自己本机的ip地址
root@student-virtual-machine:/usr/local/hadoop-2.7.6/etc/hadoop# vim yarn-site.xml


<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux_services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.100.136</value>
</property>
</configuration>

 自己创建我们配置文件中需要创建的目录：

root@student-virtual-machine:/usr/local/hadoop-2.7.6# mkdir tmp
root@student-virtual-machine:/usr/local/hadoop-2.7.6# ls
bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share  tmp
root@student-virtual-machine:/usr/local/hadoop-2.7.6# cd tmp
root@student-virtual-machine:/usr/local/hadoop-2.7.6/tmp# mkdir name
root@student-virtual-machine:/usr/local/hadoop-2.7.6/tmp# ls
name
root@student-virtual-machine:/usr/local/hadoop-2.7.6/tmp# mkdir data
root@student-virtual-machine:/usr/local/hadoop-2.7.6/tmp# ls
data  name
root@student-virtual-machine:/usr/local/hadoop-2.7.6/tmp#

 

格式化：hdfs namenode -format

看见Exiting with status 0则表示格式化成功

root@student-virtual-machine:/usr/local# cd hadoop-2.7.6/
root@student-virtual-machine:/usr/local/hadoop-2.7.6# cd bin
root@student-virtual-machine:/usr/local/hadoop-2.7.6/bin# ls


container-executor  hadoop  hadoop.cmd  hdfs  hdfs.cmd  mapred  mapred.cmd  rcc  test-container-executor  yarn  yarn.cmd


root@student-virtual-machine:/usr/local/hadoop-2.7.6/bin# hdfs namenode -format


20/12/20 15:12:54 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = student-virtual-machine/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.6
STARTUP_MSG:   classpath = /usr/local/hadoop-2.7.6/etc/hadoop:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/hadoop-annotations-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/hadoop-auth-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jsch-0.1.54.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/hadoop-nfs-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/hadoop-common-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/common/hadoop-common-2.7.6-tests.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/hadoop-hdfs-2.7.6-tests.jar:/usr/local/hadoop-2.7.6/share/hadoop/hdfs/hadoop-hdfs-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-registry-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-common-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-api-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-client-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/yarn/hadoop-yarn-common-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar:/usr/local/hadoop-2.7.6/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.6.jar:/usr/local/hadoop-2.7.6/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r 085099c66cf28be31604560c376fa282e69282b8; compiled by 'kshvachk' on 2018-04-18T01:33Z
STARTUP_MSG:   java = 1.8.0_261
************************************************************/
20/12/20 15:12:54 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
20/12/20 15:12:54 INFO namenode.NameNode: createNameNode [-format]
20/12/20 15:12:56 WARN common.Util: Path /usr/local/hadoop-2.7.6/tmp/name should be specified as a URI in configuration files. Please update hdfs configuration.
20/12/20 15:12:56 WARN common.Util: Path /usr/local/hadoop-2.7.6/tmp/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-ba84cb99-3b3a-45bb-8ef8-f44ca8fd18d2
20/12/20 15:12:56 INFO namenode.FSNamesystem: No KeyProvider found.
20/12/20 15:12:56 INFO namenode.FSNamesystem: fsLock is fair: true
20/12/20 15:12:56 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
20/12/20 15:12:56 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
20/12/20 15:12:56 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
20/12/20 15:12:56 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
20/12/20 15:12:56 INFO blockmanagement.BlockManager: The block deletion will start around 2020 Dec 20 15:12:56
20/12/20 15:12:56 INFO util.GSet: Computing capacity for map BlocksMap
20/12/20 15:12:56 INFO util.GSet: VM type       = 64-bit
20/12/20 15:12:56 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
20/12/20 15:12:56 INFO util.GSet: capacity      = 2^21 = 2097152 entries
20/12/20 15:12:56 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
20/12/20 15:12:56 INFO blockmanagement.BlockManager: defaultReplication         = 1
20/12/20 15:12:56 INFO blockmanagement.BlockManager: maxReplication             = 512
20/12/20 15:12:56 INFO blockmanagement.BlockManager: minReplication             = 1
20/12/20 15:12:56 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
20/12/20 15:12:56 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
20/12/20 15:12:56 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
20/12/20 15:12:56 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
20/12/20 15:12:57 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
20/12/20 15:12:57 INFO namenode.FSNamesystem: supergroup          = supergroup
20/12/20 15:12:57 INFO namenode.FSNamesystem: isPermissionEnabled = true
20/12/20 15:12:57 INFO namenode.FSNamesystem: HA Enabled: false
20/12/20 15:12:57 INFO namenode.FSNamesystem: Append Enabled: true
20/12/20 15:12:57 INFO util.GSet: Computing capacity for map INodeMap
20/12/20 15:12:58 INFO util.GSet: VM type       = 64-bit
20/12/20 15:12:58 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
20/12/20 15:12:58 INFO util.GSet: capacity      = 2^20 = 1048576 entries
20/12/20 15:12:58 INFO namenode.FSDirectory: ACLs enabled? false
20/12/20 15:12:58 INFO namenode.FSDirectory: XAttrs enabled? true
20/12/20 15:12:58 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
20/12/20 15:12:58 INFO namenode.NameNode: Caching file names occuring more than 10 times
20/12/20 15:12:58 INFO util.GSet: Computing capacity for map cachedBlocks
20/12/20 15:12:58 INFO util.GSet: VM type       = 64-bit
20/12/20 15:12:58 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
20/12/20 15:12:58 INFO util.GSet: capacity      = 2^18 = 262144 entries
20/12/20 15:12:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
20/12/20 15:12:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
20/12/20 15:12:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
20/12/20 15:12:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
20/12/20 15:12:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
20/12/20 15:12:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
20/12/20 15:12:58 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
20/12/20 15:12:58 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
20/12/20 15:12:58 INFO util.GSet: Computing capacity for map NameNodeRetryCache
20/12/20 15:12:58 INFO util.GSet: VM type       = 64-bit
20/12/20 15:12:58 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
20/12/20 15:12:58 INFO util.GSet: capacity      = 2^15 = 32768 entries
20/12/20 15:12:58 INFO namenode.FSImage: Allocated new BlockPoolId: BP-170360982-127.0.1.1-1608448378360
20/12/20 15:12:58 INFO common.Storage: Storage directory /usr/local/hadoop-2.7.6/tmp/name has been successfully formatted.
20/12/20 15:12:58 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop-2.7.6/tmp/name/current/fsimage.ckpt_0000000000000000000 using no compression
20/12/20 15:12:58 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop-2.7.6/tmp/name/current/fsimage.ckpt_0000000000000000000 of size 320 bytes saved in 0 seconds.
20/12/20 15:12:58 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/12/20 15:12:58 INFO util.ExitUtil: Exiting with status 0
20/12/20 15:12:58 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at student-virtual-machine/127.0.1.1
************************************************************/


root@student-virtual-machine:/usr/local/hadoop-2.7.6/bin# start-all.sh


This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop-2.7.6/logs/hadoop-root-namenode-student-virtual-machine.out
localhost: starting datanode, logging to /usr/local/hadoop-2.7.6/logs/hadoop-root-datanode-student-virtual-machine.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:6D7w/Y6900m3laVWcsaPMDrdhVpyEuqRUrhj7Fum8lE.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.7.6/logs/hadoop-root-secondarynamenode-student-virtual-machine.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.7.6/logs/yarn-root-resourcemanager-student-virtual-machine.out
localhost: starting nodemanager, logging to /usr/local/hadoop-2.7.6/logs/yarn-root-nodemanager-student-virtual-machine.out




root@student-virtual-machine:/usr/local/hadoop-2.7.6/bin# jps

25267 NodeManager
24659 DataNode
24935 SecondaryNameNode
25545 Jps
24523 NameNode
25132 ResourceManager
root@student-virtual-machine:/usr/local/hadoop-2.7.6/bin# 
root@student-virtual-machine:/usr/local/hadoop-2.7.6/bin#

可以看到我们的节点都启动了

进入浏览器测试HDFS和yarn,在浏览器中分别输入http://192.168.100.136:50070,http://192.168.100.136:8088会看到下面两个界面则表示hadoop伪分布模式搭建成功

 

 

败北L
关注
专栏目录
Hadoop初入门的坑
jiayoudeBOKE的博客
01-06 6523
这几天初入门hadoop，测试一些demo遇到了许多问题，有些问题想了好久才找到办法解决，因此写下这篇入坑小博客，学习编程，解决问题是我们的核心，所以我们遇到难题应该兴奋并解决之。以后我要写多点博客，记录编程的点点滴滴，愿与各位共勉之。下面，正题。 ！！！hadoop中遇到问题一定要找相应的日志！！！ 一、环境 1、一个linux系统(Centos7)的服务器(不是虚拟机)，下面贴出/e
Hadoop伪分布模式安装
weixin_44023300的博客
05-20 337
启动Hadoop支持三种模式操作，分别是单机模式操作、伪分布模式操作和全分布模式操作。 单机模式操作是Hadoop的默认操作模式，当首次解压Hadoop的源码包时，Hadoop无法了解硬件安装环境，会保守地选择最小配置，即单机模式。该模式主要用于开发调试MapReduce程序的应用逻辑，而不会和守护进程交互，避免增加额外的复杂性。 伪分布模式操作是指在“单节点集群”上运行Hadoop，其中所有...
hadoop2.7 伪分布
杨航的专栏
03-05 658
hadoop2.7发布，这一版不太适合用于生产环境，但是并不影响学习：由于hadoop安装方式有三种，并且三种安装方式都可以在前面的基础上继续配置，分别是： 本地模式伪分布分布式 ############################################### 1.准备 安装jdk1.7参考 linux(ubuntu)安装Java jdk环境变量
hadoop问题记录
ZHI_YUE的博客
04-30 892
22/06/13 22:13:07 INFO blockmanagement.BlockManager: The block deletion will start around 2022 6月 13 22:13:07。再反复观察/opt/hadoop-2.10.2/logs/hadoop-zhiyue-namenode-zhiyue-virtual-machine.log日志后使用教材修改方式。注：运行后仍然报该错，但jps出现datanode，可正常使用hdfs命令。运行java程序报如下错。
嵌入式LINUX（C语言编程）家目录与根目录的解析，shell编程格式，常用命令与解释
勾栏听曲_0的博客
06-19 1590
关于home目录与家目录，有一点要注意，正如下面这张图片所示，在根目录下执行“ｌｓ＂命令，可以看到一个叫home的目录，但它并不是用户的家目录，只能说是一个存储了所有用户家目录的目录。 当我们进入home目录可以看到如下 我现在登录的用户名是student，在home目录中有一个student目录，这个才是用户是student的家目录，如我现在登录的是student，那我就只能在／home／student目录下进行操作。shell命令基本格式如下： ＄：　　　　普通用户命令开始提示符 command：　
bin/hdfs dfs -mkdir xmkdir
hzdxw的专栏
06-17 3017
bin/hdfs dfs -mkdir /xmkdir
大数据之hadoop伪分布模式启动.docx
11-24
大数据之Hadoop伪分布模式启动 Hadoop伪分布模式是一种特殊的Hadoop集群模式，在这种模式下，所有Hadoop服务都运行在同一个节点上，模拟一个完整的Hadoop集群环境。这种模式非常适合开发、测试和学习Hadoop。 格式...
Linux下部署Hadoop伪分布模式
01-20
　第一步：配置ssh本地登录证书（虽然为伪分布模式，Hadoop依然会使用SSH进行通信）   [hadoop@promote ~]$ which ssh /usr/bin/ssh [hadoop@promote ~]$ which ssh-keygen /usr/bin/...
hadoop伪分布模式搭建(详细步骤）_hadoop伪分布式搭建全过程-CSDN博客.mhtml
最新发布
03-26
hadoop伪分布模式搭建(详细步骤）_hadoop伪分布式搭建全过程-CSDN博客.mhtml
hadoop hdfs记录踩到的坑
m0_46455711的博客
05-14 5396
研究了几天终于将hdfs的java api调用搞通了，其中的艰辛一度让我想要放弃，但最终让我坚持了下来。这几天的经验，不能不说是宝贵的，故记录下来，以防以后遗忘。我用的是版本2.10.0，你要问我为啥选择这个版本，我的回答是我也不知道，只知道官网上的下载列表第一个就是它。 1、下载 wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz 2、安装 我是采用Linux系统来安装的（windo
Hadoop常见重要命令行操作及命令作用
热门推荐
吃果冻不吐果冻皮
06-16 1万+
关于Hadoop[root@master ~]# hadoop --help Usage: hadoop [--config confdir] COMMAND where COMMAND is one of: fs run a generic filesystem user client version print
Hadoop伪分布模式搭建及启动
myloveprogrmming的博客
10-07 1672
伪分布等同于完全分布只不过它只有一个结点 一JDK 在虚拟机中安装gdk 1在windows下下载好，在/home下建一个文件夹downloads并把windows下载的jdk放到这个文件夹中 2把这个压缩解开 tar -xzvf 压缩包名 3在根目录下创建一个目录/soft 把这个解压缩得到的文件移到这里 4创建一个符号链接  ln -s /soft/jdk1.8.0_65 /sof...
Hadoop 实战之运行PutMerge
xiaotom5的专栏
10-17 1万+
大家好，今天给大家介绍一下Hadoop的文件系统HDFS，HDFS是一种文件系统，存储着Hadoop应用将要处理的数据，类似于普通的Unix和linux文件系统，不同的是他是实现了google的GFS文件系统的思想，是适用于大规模分布式数据处理相关应用的、可扩展的分布式文件系统。它有以下优点： 1、在分布式存储中，经常会出现节点失效的情况，HDFS可以持续监视，错误检查，容错处理，自动恢复；
Hadoop Balancer运行速度优化
勤能补拙
10-20 1311
1.修改dfs.datanode.max.transfer.threads = 4096 (如果运行hbase的话建议为16384)，指定用于在DataNode间传输block数据的最大线程数，老版本的对应参数为dfs.datanode.max.xcievers2.修改dfs.datanode.balance.bandwidthPerSec = 31457280 ,指定DataNode用于bala
大数据面试简答题 (一) - hadoop
hongchenshijie的博客
11-08 1941
1.怎么理解分布式？ 不同的业务模块部署在不同的服务器上,或者同一个业务模块分拆多个子业务，部署在不同的服务器上，解决高并发的问题。 2.hadoop 的组成部分 HDFS 分布式文件存储系统 管理者：NameNode 工作者：DataNode 辅助者：SecondayNameNode MapReduce 分布式离线计算框架 Yarn Hadoop资源调度器 管理者：R...
Hadoop 参数配置优化
系统运维
01-14 172
主要参照官方给出的配置指南进行的。 hadoop.tmp.dir 默认值：/tmp说明：尽量手动配置这个选项，否则的话都默认存在了里系统的默认临时文件/tmp里。并且手动配置的时候，如果服务器是多磁盘的，每个磁盘都设置一个临时文件目录，这样便于mapreduce或者hdfs等使用的时候提高磁盘IO效率。 fs.trash.interval 默认值：0说明：这个是开启hdfs文件...
深入理解Hadoop YARN中的Container概念
Mac Track
12-22 7535
在学习Hadoop  YARN—Hadoop 2.0新引入的通用资源管理系统过程中，总会遇到Container这一概念，由于中文资料的缺乏，很多人对Container这一概念仍非常的模糊。它与Linux Container是什么关系，它是否能像Linux Container那样为任务提供一个隔离环境?它代表计算资源，还是仅仅是一个任务处理进程？本文将尝试介绍Container这一概念。
Hadoop 伪分布模式安装配置详解
配置文件的修改是Hadoop伪分布模式安装的关键步骤。主要涉及四个文件，它们都位于$HADOOP_HOME/conf目录下： 1. `hadoop-env.sh`：在这里设置JAVA_HOME环境变量，确保Hadoop能找到Java的安装位置，通常去掉注释并...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值