前提条件
拥有CentOS7服务器版环境
软件版本说明
jdk1.8
hadoop2.7.3
虚拟机环境准备
通网络
能ping通外网，例如：
ping baidu.com
如果ping不通，可以修改如下文件：
vi /etc/sysconfig/network-scripts/ifcfg-ens33
将ONBOOT=no改为ONBOOT=yes
重启网络或重启机器
重启网络
systemctl restart network
重启机器
reboot
修改主机名
将主机名从localhost改为node1，操作过程如下：
[root@localhost ~]# hostname localhost.localdomain [root@localhost ~]# hostnamectl set-hostname node1 [root@localhost ~]# hostname node1 未重启时主机名未变 [root@localhost ~]# reboot ... Connecting to 192.168.193.140:22... Connection established. To escape to local shell, press 'Ctrl+Alt+]'. WARNING! The remote SSH server rejected X11 forwarding request. Last login: Tue Mar 8 00:55:08 2022 from 192.168.193.1 [root@node1 ~]# 重启后，命令行中的主机名改变了
新增普通用户
因为root用户权限太高，误操作可能造成一些麻烦，一般情况下，我们不会用root来安装hadoop，所以我们需要创建普通用户来安装hadoop。创建新用户方法如下：
创建新用户，例如用户名为：hadoop，并设置新用户密码，重复设置2次，看不到输入的密码，这是linux的安全机制，输入即可。
[root@node1 ~]# adduser hadoop [root@node1 ~]# passwd hadoop Changing password for user hadoop. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully.
给普通用户添加sudo执行权限
[root@node1 ~]# chmod -v u+w /etc/sudoers mode of ‘/etc/sudoers’ changed from 0440 (r--r-----) to 0640 (rw-r-----) [root@node1 ~]# vim /etc/sudoers 在%wheel ALL=(ALL) ALL一行下面添加如下语句： hadoop ALL=(ALL) ALL [root@node1 ~]# chmod -v u-w /etc/sudoers mode of ‘/etc/sudoers’ changed from 0640 (rw-r-----) to 0440 (r--r-----)
静态IP（非必要）
因为电脑连接网络不同，CentOS的IP可能会变化，可以设置静态IP来解决。
网关查询：
Vmware-->编辑-->虚拟机网络编辑器
注意：这里查询到网关为192.168.193.2，以自己实际查到的网关为准。
设置静态ip
[root@node1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
修改或添加如下几项：
BOOTPROTO=static IPADDR=192.168.193.140 GATEWAY=192.168.193.2 DNS1=192.168.193.2
注意：
1.GATEWAY和DNS1的值都填写上一步在vmware中实际查到的网关地址。
2.IPADDR为设置的静态IP地址，要与查到的网关处于同一网络，查到的网关所处的网段为192.168.193，IP地址不要与网关一致，最后一位建议设置为128-255之间，这里IP最后一位设置为140。
关闭防火墙
查看防火墙状态
[root@node1 ~]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) [root@node1 ~]#
inactive (dead)为关闭状态
如果是active (running)，为开启状态，可以用如下命令关闭防火墙
[root@node1 ~]# systemctl stop firewalld
安装hadoop伪分布式
登录普通用户
使用普通用户登录CentOS7，这里登录的普通用户为hadoop
[hadoop@node1 ~]$
映射IP
[hadoop@node1 ~]$ sudo vim /etc/hosts
文件末尾，添加如下内容：
192.168.193.140 node1
目录准备
创建2个目录：installfile和soft，installfile存放软件安装包，soft是软件的安装目录。
[hadoop@node1 ~]$ mkdir installfile [hadoop@node1 ~]$ mkdir soft [hadoop@node1 soft]$ ls soft installfile
安装配置jdk
官网下载jdk安装包：jdk-8u271-linux-x64.tar.gz，并将安装包上传到Linux的installfile目录。
进入安装包所在目录
cd ~/installfile
解压安装包
[hadoop@node1 installfile]$ tar -zxvf jdk-8u271-linux-x64.tar.gz -C ~/soft/
创建软链接
切换到解压目录 [hadoop@node1 installfile]$ cd ~/soft/ [hadoop@node1 soft]$ ls jdk1.8.0_271 创建软链接 [hadoop@node1 soft]$ ln -s jdk1.8.0_271 jdk 查看软链接 [hadoop@node1 soft]$ ls jdk jdk1.8.0_271
安装nano命令（非必要，因为个人习惯使用nano编辑命令）
sudo yum install nano -y
nano命令的功能和vi命令一样（如果不习惯用nano，后面可以用vi代替nano）。
nano的使用方法：nano 文件名，输入相关内容，保存按Ctrl+O，回车确认文件名，退出按Ctrl+X
配置环境变量
这里在.bashrc文件配置环境变量，也可以配置在/etc/profile文件。
[hadoop@node1 soft]$ nano ~/.bashrc
在文件末尾添加如下语句
export JAVA_HOME=~/soft/jdk
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
让配置立即生效
[hadoop@node1 soft]$ source ~/.bashrc
验证
[hadoop@node1 soft]$ java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
能看到输出版本号即为安装配置成功。
安装配置hadoop
下载并上传安装包到Linux目录中，例如：~/installfile
[hadoop@node1 installfile]$ cd ../installfile/
解压
[hadoop@node1 installfile]$ tar -zxvf hadoop-2.7.3.tar.gz -C ~/soft/
创建软链接
[hadoop@node1 installfile]$ cd ~/soft/ [hadoop@node1 soft]$ ls hadoop-2.7.3 jdk jdk1.8.0_271 创建软链接 [hadoop@node1 soft]$ ln -s hadoop-2.7.3 hadoop [hadoop@node1 soft]$ ls hadoop hadoop-2.7.3 jdk jdk1.8.0_271
配置环境变量
nano ~/.bashrc
文件末尾添加如下内容：
export HADOOP_HOME=~/soft/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
让环境变量立即生效
source ~/.bashrc
验证
[hadoop@node1 soft]$ hadoop version Hadoop 2.7.3 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff Compiled by root on 2016-08-18T01:41Z Compiled with protoc 2.5.0 From source with checksum 2e4ce5f957ea4db193bce3734ff29ff4 This command was run using /home/hadoop/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar
能看到hadoop版本号，说明环境变量设置成功。
设置免密登录，执行ssh-keygen -t rsa命令后，再连续敲击3次回车
[hadoop@node1 soft]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: SHA256:sXZw/coSYm19gGSLdOLQGeskm+vPVr6lXENvmmQ8eeo hadoop@node1 The key's randomart image is: +---[RSA 2048]----+ | ..+o+ | | +oB + | | . B + o | | * * . o | | o S = o o | | + +.= = | | . o. % + | | . ....B O | | .oo +oE | +----[SHA256]-----+
查看生成的密钥对
[hadoop@node1 soft]$ ls ~/.ssh/ id_rsa id_rsa.pub
追加公钥，执行命令后，根据提示输入yes再次回车
[hadoop@node1 soft]$ ssh-copy-id node1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub" The authenticity of host 'node1 (192.168.193.140)' can't be established. ECDSA key fingerprint is SHA256:r4LfTp6UJiZJbbxCrCaF8ouB1XEySe9c1/5d+7a39+E. ECDSA key fingerprint is MD5:08:e4:65:b1:9a:16:5d:fd:f8:ac:13:fc:4c:88:f4:8e. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@node1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'node1'" and check to make sure that only the key(s) you wanted were added. [hadoop@node1 soft]$
查看生成的认证文件authorized_keys
[hadoop@node1 soft]$ ls ~/.ssh/ authorized_keys id_rsa id_rsa.pub known_hosts 查看认证文件内容authorized_keys [hadoop@node1 soft]$ cat ~/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuG6EHCrL93O355sr7dkLO8S75QqizvCXNQztwY0F2mrVmUZNk9BleNRE1jAirOQccFTR11SCGIhwa1klPYvGwAtY66YG+y47tsKkLc9VHoIvfpwjG3MzSwuAJhhsnv2UOIGX8kkP0/cSVT1YyfM3cJO0OzN9gB4vTE/0VUJRgGk43qO3v84kWIPTZrJlzr8wz5GSD3quteRea6Gxv7EPksWfP18sZDsynvYZ8zvXzmiv1Qf1g5Pcc52RhF0ghsZoVHB03KAPX72d4N33budSy7jnRTz81NSqnTZ8ajLOSIQVxeFk6tJwHOLW/uDQGy124PaYkiElrVyHesxJbC0Vl hadoop@node1 查看公钥 [hadoop@node1 soft]$ cat ~/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuG6EHCrL93O355sr7dkLO8S75QqizvCXNQztwY0F2mrVmUZNk9BleNRE1jAirOQccFTR11SCGIhwa1klPYvGwAtY66YG+y47tsKkLc9VHoIvfpwjG3MzSwuAJhhsnv2UOIGX8kkP0/cSVT1YyfM3cJO0OzN9gB4vTE/0VUJRgGk43qO3v84kWIPTZrJlzr8wz5GSD3quteRea6Gxv7EPksWfP18sZDsynvYZ8zvXzmiv1Qf1g5Pcc52RhF0ghsZoVHB03KAPX72d4N33budSy7jnRTz81NSqnTZ8ajLOSIQVxeFk6tJwHOLW/uDQGy124PaYkiElrVyHesxJbC0Vl hadoop@node1 可发现，认证文件的内容和公钥一致。
验证免密登录
ssh登录到node1,不需要输入密码，注意路径的变化：由soft目录切换到了~目录，执行exit退出ssh登录 [hadoop@node1 soft]$ ssh node1 Last login: Tue Mar 8 05:54:02 2022 from 192.168.193.1 [hadoop@node1 ~]$ exit logout Connection to node1 closed. [hadoop@node1 soft]$ 也可以ssh ip [hadoop@node1 soft]$ ssh 192.168.193.140 Last login: Tue Mar 8 06:16:58 2022 from node1 [hadoop@node1 ~]$ exit logout Connection to 192.168.193.140 closed.
配置hadoop伪分布式
进入hadoop配置目录
cd ${HADOOP_HOME}/etc/hadoop
[hadoop@node1 soft]$ cd ${HADOOP_HOME}/etc/hadoop
[hadoop@node1 hadoop]$ ls
capacity-scheduler.xml hadoop-policy.xml kms-log4j.properties ssl-client.xml.example
configuration.xsl hdfs-site.xml kms-site.xml ssl-server.xml.example
container-executor.cfg httpfs-env.sh log4j.properties yarn-env.cmd
core-site.xml httpfs-log4j.properties mapred-env.cmd yarn-env.sh
hadoop-env.cmd httpfs-signature.secret mapred-env.sh yarn-site.xml
hadoop-env.sh httpfs-site.xml mapred-queues.xml.template
hadoop-metrics2.properties kms-acls.xml mapred-site.xml.template
hadoop-metrics.properties kms-env.sh slaves
配置hadoop-env.sh
[hadoop@node1 hadoop]$ nano hadoop-env.sh
配置JAVA_HOME
export JAVA_HOME=/home/hadoop/soft/jdk
配置core-site.xml, 在<configuration>与</configuration>之间添加如下配置
<property> <name>fs.defaultFS</name> <value>hdfs://node1:8020</value> <!-- 以上主机名node1要按实际情况修改 --> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/soft/hadoop/tmp</value> </property>
配置hdfs-site.xml
同样在<configuration>与</configuration>之间添加配置内容如下：
<property> <name>dfs.replication</name> <value>1</value> </property>
复制模板文件mapred-site.xml.template为mapred-site.xml
[hadoop@node1 hadoop]$ cp mapred-site.xml.template mapred-site.xml
配置mapred-site.xml
同样在<configuration>与</configuration>之间添加配置内容如下：
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
以上设置了mapreduce运行在yarn框架之上。
配置yarn-site.xml
同样在<configuration>与</configuration>之间添加配置内容如下：
<property> <name>yarn.resourcemanager.hostname</name> <value>node1</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>
配置slaves
配置从节点所在的机器
nano slaves
将localhost修改为主机名，例如: node1
node1
格式化文件系统
hdfs namenode -format
[hadoop@node1 hadoop]$ hdfs namenode -format 22/03/08 06:31:01 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = node1/192.168.193.140 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /home/hadoop/soft/hadoop-2.7.3/etc/hadoop:/home/hadoop/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/soft/hadoop- ... 2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/home/hadoop/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/home/hadoop/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/home/hadoop/soft/hadoop/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z STARTUP_MSG: java = 1.8.0_271 ************************************************************/ 22/03/08 06:31:01 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 22/03/08 06:31:01 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-911f3db2-a66f-4748-a2bd-bb427d00f2e4 22/03/08 06:31:01 INFO namenode.FSNamesystem: No KeyProvider found. 22/03/08 06:31:01 INFO namenode.FSNamesystem: fsLock is fair:true 22/03/08 06:31:01 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 22/03/08 06:31:01 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 22/03/08 06:31:01 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 22/03/08 06:31:01 INFO blockmanagement.BlockManager: The block deletion will start around 2022 Mar 08 06:31:01 22/03/08 06:31:01 INFO util.GSet: Computing capacity for map BlocksMap 22/03/08 06:31:01 INFO util.GSet: VM type = 64-bit 22/03/08 06:31:01 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB 22/03/08 06:31:01 INFO util.GSet: capacity = 2^21 = 2097152 entries 22/03/08 06:31:02 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 22/03/08 06:31:02 INFO blockmanagement.BlockManager: defaultReplication = 1 22/03/08 06:31:02 INFO blockmanagement.BlockManager: maxReplication = 512 22/03/08 06:31:02 INFO blockmanagement.BlockManager: minReplication = 1 22/03/08 06:31:02 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 22/03/08 06:31:02 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 22/03/08 06:31:02 INFO blockmanagement.BlockManager: encryptDataTransfer = false 22/03/08 06:31:02 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 22/03/08 06:31:02 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE) 22/03/08 06:31:02 INFO namenode.FSNamesystem: supergroup = supergroup 22/03/08 06:31:02 INFO namenode.FSNamesystem: isPermissionEnabled = true 22/03/08 06:31:02 INFO namenode.FSNamesystem: HA Enabled: false 22/03/08 06:31:02 INFO namenode.FSNamesystem: Append Enabled: true 22/03/08 06:31:02 INFO util.GSet: Computing capacity for map INodeMap 22/03/08 06:31:02 INFO util.GSet: VM type = 64-bit 22/03/08 06:31:02 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB 22/03/08 06:31:02 INFO util.GSet: capacity = 2^20 = 1048576 entries 22/03/08 06:31:02 INFO namenode.FSDirectory: ACLs enabled? false 22/03/08 06:31:02 INFO namenode.FSDirectory: XAttrs enabled? true 22/03/08 06:31:02 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 22/03/08 06:31:02 INFO namenode.NameNode: Caching file names occuring more than 10 times 22/03/08 06:31:02 INFO util.GSet: Computing capacity for map cachedBlocks 22/03/08 06:31:02 INFO util.GSet: VM type = 64-bit 22/03/08 06:31:02 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB 22/03/08 06:31:02 INFO util.GSet: capacity = 2^18 = 262144 entries 22/03/08 06:31:02 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 22/03/08 06:31:02 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 22/03/08 06:31:02 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 22/03/08 06:31:02 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 22/03/08 06:31:02 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 22/03/08 06:31:02 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 22/03/08 06:31:02 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 22/03/08 06:31:02 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 22/03/08 06:31:02 INFO util.GSet: Computing capacity for map NameNodeRetryCache 22/03/08 06:31:02 INFO util.GSet: VM type = 64-bit 22/03/08 06:31:02 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB 22/03/08 06:31:02 INFO util.GSet: capacity = 2^15 = 32768 entries 22/03/08 06:31:02 INFO namenode.FSImage: Allocated new BlockPoolId: BP-568726822-192.168.193.140-1646692262515 22/03/08 06:31:02 INFO common.Storage: Storage directory /home/hadoop/soft/hadoop/tmp/dfs/name has been successfully formatted. 22/03/08 06:31:02 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/soft/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 22/03/08 06:31:02 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/soft/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds. 22/03/08 06:31:02 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 22/03/08 06:31:02 INFO util.ExitUtil: Exiting with status 0 22/03/08 06:31:02 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.193.140 ************************************************************/ [hadoop@node1 hadoop]$
若能看到Exiting with status 0，则格式化成功
注意：格式化只需要做一次，格式化成功后，以后就不能再次格式化了。
启动hdfs
start-dfs.sh
验证
jps查看hdfs进程，hdfs进程包括：NameNode、DataNode、SecondaryNameNode
[hadoop@node1 hadoop]$ jps 1989 NameNode 2424 Jps 2300 SecondaryNameNode 2125 DataNode
如果少进程，可以切换到$HADOOP_HOME/logs目录，查看对应进程的log文件排查原因，例如：缺少DataNode，应该查看的DataNode的log文件，查看log报的错误、警告、异常信息（Error、Warning、Exception），找到少进程的原因，并解决。
浏览器验证ip:50070，例如：192.168.193.140:50070
启动yarn
start-yarn.sh
验证
jps查yarn进程，yarn进程包括：ResourceManager和NodeManager
[hadoop@node1 hadoop]$ jps 2706 NodeManager 1989 NameNode 2885 Jps 2475 ResourceManager 2300 SecondaryNameNode 2125 DataNode
如果少进程，同样可查看相关log文件来排查。
浏览器访问ip:8088 例如：192.168.193.140:8088
完成！enjoy it!
1001
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
