目录
Linux准备
openEuler24.03 LTS简介
Linux选择国产的openEuler24.03 LTS。
openEuler 24.03 LTS 是华为捐赠给开放原子开源基金会的开源操作系统 openEuler 的长期支持版本，于2024年6月6日正式发布。作为首个AI原生开源操作系统，其聚焦于服务器、云计算、边缘计算及嵌入式设备等数字基础设施领域。
下载openEuler24.03 LTS
https://www.openeuler.org/en/download/
下载openEuler24.03 LTS SP1的Offline Standard ISO文件：openEuler-24.03-LTS-SP1-x86_64-dvd.iso
安装openEuler24.03 LTS
创建一台虚拟机名字为node2的机器，然后安装openEuler24.03 LTS SP1，可参考：Vmware下安装openEuler24.03 LTS
Linux基本设置
关闭及禁用防火墙
[root@localhost ~]# systemctl stop firewalld [root@localhost ~]# systemctl disable firewalld
修改主机名
修改主机名为node2
# 修改主机名 [root@localhost ~]# hostnamectl set-hostname node2 # 重启 [root@localhost ~]# reboot
重启后，重新用远程工具连接，看到显示的主机名已经变为node2
[root@node2 ~]#
静态ip
默认为DHCP，ip可能会变化，ip变化会带来不必要的麻烦，所以需要将ip固定下来方便使用。
[root@node2 ~]# cd /etc/sysconfig/network-scripts/ [root@node2 network-scripts]# ls ifcfg-ens33 [root@node2 network-scripts]# vim ifcfg-ens33
修改内容如下
# 修改
BOOTPROTO=static
# 添加
IPADDR=192.168.193.132
NETMASK=255.255.255.0
GATEWAY=192.168.193.2
DNS1=192.168.193.2
DNS2=114.114.114.114这里设置的固定IP为192.168.193.132。注意：IPADDR、GATEWAY、DNS，使用192.168.193.*的网段要与Vmware查询到的NAT网络所在的网段一致，请根据实际情况修改网段值，网段查询方法：打开Vmware，文件-->虚拟网络编辑器。
重启生效
reboot
映射主机名
修改/etc/hosts
[root@node2 ~]$ vim /etc/hosts
末尾添加如下内容
192.168.193.132 node2
192.168.193.133 node3
192.168.193.134 node4注意：ip和主机名，请根据实际情况修改。集群规划用到node3和node4，提前写入node3和node4 映射信息。
创建普通用户
因为root用户权限太高，误操作可能会造成不可挽回的损失，所以需要新建一个普通用户来进行后续大数据环境操作。例如：这里创建一个名为liang的普通用户，密码也是liang，注意：用户名和密码请根据实际需要修改。命令如下：
useradd liang passwd liang
操作过程
[root@node2 ~]# useradd liang [root@node2 ~]# passwd liang 更改用户 liang 的密码 。 新的密码： 无效的密码： 密码少于 8 个字符 重新输入新的密码： passwd：所有的身份验证令牌已经成功更新。
虽然提示无效密码，但已经更新成功。
给新用户添加sudo权限
修改/etc/sudoers文件
vim /etc/sudoers
在%wheel这行下面添加如下一行
liang ALL=(ALL) NOPASSWD:ALL注意：liang是用户名，需要根据实际情况修改。
保存按Esc退出编辑模式，再按:wq!
目录准备
目录规划：
1.把软件安装包放在/opt/software目录；
2.把可自定义安装目录的软件安装在/opt/module目录。
注意：规划的目录可以根据实际需要修改。
创建目录及修改权限
[root@node2 ~]# mkdir /opt/module [root@node2 ~]# mkdir /opt/software [root@node2 ~]# chown liang:liang /opt/module [root@node2 ~]# chown liang:liang /opt/software
注意：如果普通用户不是liang，chown命令的liang需要根据实际情况修改。
克隆主机
克隆node2机器得到node3和node4
操作克隆node2得到node3
克隆方法：在node2为关机状态下，点击 虚拟机-->管理-->克隆，克隆类型选择创建完整克隆，根据提示完成克隆。
设置静态ip
打开node3机器
[root@node2 ~]# cd /etc/sysconfig/network-scripts/ [root@node2 network-scripts]# ls ifcfg-ens33 [root@node2 network-scripts]# vim ifcfg-ens33
将ip地址改为
192.168.193.133修改主机名为node3
# 修改主机名 [root@node2 ~]$ hostnamectl set-hostname node3 # 查看主机名 [root@node2 ~]$ hostname node3 # 重启机器 [root@node2 ~]$ reboot
登录普通用户liang验证主机名和ip地址，确实已经为node3
[liang@node3 ~]$ hostname node3 [liang@node3 ~]$ ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.193.133 netmask 255.255.255.0 broadcast 192.168.193.255 inet6 fe80::20c:29ff:feaa:b060 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:aa:b0:60 txqueuelen 1000 (Ethernet) RX packets 100 bytes 12934 (12.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 106 bytes 15512 (15.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [liang@node3 ~]$
操作克隆node2得到node4
同样的方法，操作克隆node2得到node4，设置静态ip为192.168.193.134，修改主机名为node4。
配置机器之间免密登录
后续的安装操作都在普通用户下操作，所以需要在普通用户下设置SSH免密登录。
在node2机器操作：
登录node2普通用户（liang），执行如下命令生成密钥对
ssh-keygen -t rsa
执行命令后，连续敲击三次回车键
拷贝公钥
ssh-copy-id node2 ssh-copy-id node3 ssh-copy-id node4
执行ssh-copy-id命令后，根据提示输入yes，再输入机器登录密码
验证
从node2发起ssh登录到node3，过程中不需要登录密码为配置成功，使用exit退出免密登录。
ssh node3 exit
同样的方法，在node3、node4机器上操作。
编写分发脚本
使用rsync命令分发，可以实现增量复制，速度快。
在主目录创建bin目录
[liang@node2 ~]$ mkdir ~/bin
创建分发脚本文件xsync
[liang@node2 ~]$ vim ~/bin/xsync
内容如下
#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
echo Not Enough Arguement!
exit;
fi
#2. 遍历集群所有机器
for host in node2 node3 node4
do
echo ==================== $host ====================
#3. 遍历所有目录，挨个发送
for file in $@
do
#4. 判断文件是否存在
if [ -e $file ]
then
#5. 获取父目录
pdir=$(cd -P $(dirname $file); pwd)
#6. 获取当前文件的名称
fname=$(basename $file)
ssh $host "mkdir -p $pdir"
rsync -av $pdir/$fname $host:$pdir
else
echo $file does not exists!
fi
done
done修改权限
[hadoop@node2 ~]$ chmod +x ~/bin/xsync
添加环境变量
[liang@node2 ~]$ sudo vim /etc/profile.d/my_env.sh
添加内容
#MyShellCommand
export PATH=$PATH:/home/liang/bin让环境变量生效
[liang@node2 ~]$ source /etc/profile
测试
把xsync命令发送到node3、node4
xsync /home/liang/bin
查看node3、node4是否有收到xsync脚本。
[liang@node3 ~]$ ls bin/ xsync [liang@node4 ~]$ ls bin/ xsync
安装Java
Java是基础软件，查看Hadoop支持的Java版本
Supported Java Versions Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only) Please compile Hadoop with Java 8. Compiling Hadoop with Java 11 is not supported: Apache Hadoop from 3.0.x to 3.2.x now supports only Java 8 Apache Hadoop from 2.7.x to 2.10.x support both Java 7 and 8
看到Hadoop3.3及以上版本只支持Java8和Java11，编译只支持Java8。若使用更高版本的Java，需要做一定的适配，所以这里选择Java8。
先在node2上安装Java，然后再分发拷贝到其他机器。
下载Java
下载Java8，下载版本为：jdk-8u271-linux-x64.tar.gz，浏览器访问如下下载地址，找到并下载需要的版本：
https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html
登录node2普通用户
将jdk-8u271-linux-x64.tar.gz上传到Linux的/opt/software
[liang@node2 opt]$ ls /opt/software/ jdk-8u271-linux-x64.tar.gz
解压
[liang@node2 opt]$ cd /opt/software/ [liang@node2 software]$ ls jdk-8u271-linux-x64.tar.gz [liang@node2 software]$ tar -zxvf jdk-8u271-linux-x64.tar.gz -C /opt/module/
设置环境变量
[liang@node2 software]$ sudo vim /etc/profile.d/my_env.sh
末尾添加如下内容
#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_271
export PATH=$PATH:$JAVA_HOME/bin让环境变量生效
[liang@node2 software]$ source /etc/profile
查看版本
[liang@node2 module]$ java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
正常可以看到java version "1.8.0.271"版本号输出，如果看不到，再检查前面的步骤是否正确。
分发到其他机器
分发安装文件
/home/liang/bin/xsync /opt/module/jdk1.8.0_271
分发环境变量
sudo /home/liang/bin/xsync /etc/profile.d/my_env.sh
因为my_env.sh是root权限，所以命令前要加sudo，过程中需要根据提示输入yes及node2机器root账户的登录密码。
让环境变量立即生效，需要分别在node3、node4执行如下命令
source /etc/profile
安装Hadoop
安装配置Hadoop完全分布式
Hadoop集群规划
| 项目 | node2 | node3 | node4 |
|---|---|---|---|
| HDFS | NameNode、DataNode | DataNode | DataNode、SecondaryNameNode |
| Yarn | NodeManager | Resourcemanager、NodeManager | NodeManager |
下载hadoop
浏览器下载hadoop安装包，下载版本为hadoop-3.3.4
https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
上传hadoop安装包到Linux /opt/software
[liang@node2 opt]$ ls /opt/software/ | grep hadoop hadoop-3.3.4.tar.gz
解压
[liang@node2 opt]$ cd /opt/software/ [liang@node2 software]$ tar -zxvf hadoop-3.3.4.tar.gz -C /opt/module/
设置环境变量
[liang@node2 software]$ sudo vim /etc/profile.d/my_env.sh
文件末尾，添加如下内容
#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.3.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin让环境变量立即生效
[liang@node2 software]$ source /etc/profile
查看版本
[liang@node2 software]$ hadoop version Hadoop 3.3.4 Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb Compiled by stevel on 2022-07-29T12:32Z Compiled with protoc 3.7.1 From source with checksum fb9dd8918a7b8a5b430d61af858f6ec This command was run using /opt/module/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar
配置hadoop
配置hadoop完全分布式
进入配置文件所在目录，并查看配置文件
[liang@node2 software]$ cd $HADOOP_HOME/etc/hadoop/ [liang@node2 hadoop]$ ls capacity-scheduler.xml httpfs-env.sh mapred-site.xml configuration.xsl httpfs-log4j.properties shellprofile.d container-executor.cfg httpfs-site.xml ssl-client.xml.example core-site.xml kms-acls.xml ssl-server.xml.example hadoop-env.cmd kms-env.sh user_ec_policies.xml.template hadoop-env.sh kms-log4j.properties workers hadoop-metrics2.properties kms-site.xml yarn-env.cmd hadoop-policy.xml log4j.properties yarn-env.sh hadoop-user-functions.sh.example mapred-env.cmd yarnservice-log4j.properties hdfs-rbf-site.xml mapred-env.sh yarn-site.xml hdfs-site.xml mapred-queues.xml.template
配置core-site.xml
[liang@node2 hadoop]$ vim core-site.xml
在<configuration>和</configuration>之间添加如下内容
<!-- 指定NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://node2:8020</value>
</property>
<!-- 指定hadoop数据的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.3.4/data</value>
</property>
<!-- 配置HDFS网页登录使用的静态用户为liang -->
<property>
<name>hadoop.http.staticuser.user</name>
<value>liang</value>
</property>
<!-- 配置该liang(superUser)允许通过代理访问的主机节点 -->
<property>
<name>hadoop.proxyuser.liang.hosts</name>
<value>*</value>
</property>
<!-- 配置该liang(superUser)允许通过代理用户所属组 -->
<property>
<name>hadoop.proxyuser.liang.groups</name>
<value>*</value>
</property>
<!-- 配置该liang(superUser)允许通过代理的用户-->
<property>
<name>hadoop.proxyuser.liang.users</name>
<value>*</value>
</property>注意：如果主机名不是node2，用户名不是liang，根据实际情况修改主机名和用户名，后续的配置同样注意修改。
配置hdfs-site.xml
[liang@node2 hadoop]$ vim hdfs-site.xml
在<configuration>和</configuration>之间添加如下内容
<!-- nn web端访问地址-->
<property>
<name>dfs.namenode.http-address</name>
<value>node2:9870</value>
</property>
<!-- 2nn web端访问地址-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>node4:9868</value>
</property>
<!-- 测试环境指定HDFS副本的数量1 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>注意：副本数根据实际需要设置，生产环境副本数要大于1，例如：3。
配置mapred-site.xml
[liang@node2 hadoop]$ vim mapred-site.xml
同样在<configuration>与</configuration>之间添加配置内容如下
<!-- mapreduce运行在yarn框架之上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- 历史服务器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>node2:10020</value>
</property>
<!-- 历史服务器web端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>node2:19888</value>
</property>
配置yarn-site.xml
[liang@node2 hadoop]$ vim yarn-site.xml
同样在<configuration>与</configuration>之间添加配置内容如下
<!-- 指定ResourceManager的地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node3</value>
</property>
<!-- 指定MR走shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 环境变量的继承 -->
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<!--yarn单个容器允许分配的最大最小内存 -->
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property>
<!-- yarn容器允许管理的物理内存大小 -->
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<!-- 关闭yarn对物理内存和虚拟内存的限制检查 -->
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<!-- 开启日志聚集功能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>
<name>yarn.log.server.url</name>
<value>http://node2:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
配置workers
配置从节点所在的机器
[liang@node2 hadoop]$ vim workers
将localhost修改为如下主机名
node2
node3
node4分发到其他机器
分发安装文件到其他机器
/home/liang/bin/xsync /opt/module/hadoop-3.3.4
分发环境变量
sudo /home/liang/bin/xsync /etc/profile.d/my_env.sh
因为my_env.sh是root权限，所以命令前要加sudo，过程中需要根据提示输入node2机器root账户的登录密码。
分别让node3及node4的环境变量生效
[liang@node3 ~]$ source /etc/profile [liang@node4 ~]$ source /etc/profile
格式化文件系统
在node2操作
[liang@node2 hadoop]$ hdfs namenode -format
看到successfully formatted输出，说明格式化成功。
注意：格式化只能做一次，格式化成功后就不能再次格式化了。
启动集群
启动hdfs
在node2机器启动hdfs
[liang@node2 hadoop]$ start-dfs.sh
启动yarn
在node3机器启动yarn
[liang@node3 hadoop]$ start-yarn.sh
查看jps进程
分别在不同机器执行jps命令
[liang@node2 hadoop]$ jps 3767 DataNode 4199 NodeManager 4407 Jps 3566 NameNode [liang@node3 ~]$ jps 3555 NodeManager 3205 DataNode 3417 ResourceManager 3996 Jps [liang@node4 ~]$ jps 3555 NodeManager 3332 SecondaryNameNode 3765 Jps 3166 DataNode
访问Web UI
为了能使用主机名访问，修改Windows下的C:\Windows\System32\drivers\etc\hosts文件，添加如下映射语句
192.168.193.132 node2
192.168.193.133 node3
192.168.193.134 node4注意：根据实际情况修改ip和主机名
浏览器访问
node2:9870
浏览器访问
node3:8088
测试Hadoop
计算pi
[liang@node2 hadoop]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.4.jar pi 2 4 Number of Maps = 2 Samples per Map = 4 Wrote input for Map #0 Wrote input for Map #1 Starting Job 2025-03-18 23:15:49,010 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node3/192.168.193.133:8032 2025-03-18 23:15:49,696 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/liang/.staging/job_1742310641710_0001 2025-03-18 23:15:50,236 INFO input.FileInputFormat: Total input files to process : 2 2025-03-18 23:15:51,045 INFO mapreduce.JobSubmitter: number of splits:2 2025-03-18 23:15:51,599 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1742310641710_0001 2025-03-18 23:15:51,599 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-03-18 23:15:51,782 INFO conf.Configuration: resource-types.xml not found 2025-03-18 23:15:51,782 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-03-18 23:15:52,460 INFO impl.YarnClientImpl: Submitted application application_1742310641710_0001 2025-03-18 23:15:52,555 INFO mapreduce.Job: The url to track the job: http://node3:8088/proxy/application_1742310641710_0001/ 2025-03-18 23:15:52,556 INFO mapreduce.Job: Running job: job_1742310641710_0001 2025-03-18 23:16:04,788 INFO mapreduce.Job: Job job_1742310641710_0001 running in uber mode : false 2025-03-18 23:16:04,789 INFO mapreduce.Job: map 0% reduce 0% 2025-03-18 23:16:13,970 INFO mapreduce.Job: map 100% reduce 0% 2025-03-18 23:16:20,025 INFO mapreduce.Job: map 100% reduce 100% 2025-03-18 23:16:21,100 INFO mapreduce.Job: Job job_1742310641710_0001 completed successfully 2025-03-18 23:16:21,262 INFO mapreduce.Job: Counters: 55 File System Counters FILE: Number of bytes read=50 FILE: Number of bytes written=829296 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=522 HDFS: Number of bytes written=215 HDFS: Number of read operations=13 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=26878 Total time spent by all reduces in occupied slots (ms)=6476 Total time spent by all map tasks (ms)=13439 Total time spent by all reduce tasks (ms)=3238 Total vcore-milliseconds taken by all map tasks=13439 Total vcore-milliseconds taken by all reduce tasks=3238 Total megabyte-milliseconds taken by all map tasks=13761536 Total megabyte-milliseconds taken by all reduce tasks=3315712 Map-Reduce Framework Map input records=2 Map output records=4 Map output bytes=36 Map output materialized bytes=56 Input split bytes=286 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=56 Reduce input records=4 Reduce output records=0 Spilled Records=8 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=222 CPU time spent (ms)=2910 Physical memory (bytes) snapshot=835469312 Virtual memory (bytes) snapshot=7758372864 Total committed heap usage (bytes)=621281280 Peak Map Physical memory (bytes)=307945472 Peak Map Virtual memory (bytes)=2587164672 Peak Reduce Physical memory (bytes)=226463744 Peak Reduce Virtual memory (bytes)=2590654464 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=236 File Output Format Counters Bytes Written=97 Job Finished in 32.328 seconds Estimated value of Pi is 3.50000000000000000000 [liang@node2 hadoop]$
计算wordcount
准备输入数据
[liang@node2 ~]$ vim 1.txt [liang@node2 ~]$ cat 1.txt hello world hello hadoop [liang@node2 ~]$ hdfs dfs -put 1.txt / [liang@node2 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 1 liang supergroup 25 2025-03-18 23:17 /1.txt drwx------ - liang supergroup 0 2025-03-18 23:15 /tmp drwxr-xr-x - liang supergroup 0 2025-03-18 23:15 /user
运行wordcount程序
[liang@node2 ~]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.4.jar wordcount /1.txt /out 2025-03-18 23:18:10,177 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node3/192.168.193.133:8032 2025-03-18 23:18:11,025 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/liang/.staging/job_1742310641710_0002 2025-03-18 23:18:11,462 INFO input.FileInputFormat: Total input files to process : 1 2025-03-18 23:18:11,631 INFO mapreduce.JobSubmitter: number of splits:1 2025-03-18 23:18:11,821 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1742310641710_0002 2025-03-18 23:18:11,821 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-03-18 23:18:12,091 INFO conf.Configuration: resource-types.xml not found 2025-03-18 23:18:12,091 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-03-18 23:18:12,213 INFO impl.YarnClientImpl: Submitted application application_1742310641710_0002 2025-03-18 23:18:12,299 INFO mapreduce.Job: The url to track the job: http://node3:8088/proxy/application_1742310641710_0002/ 2025-03-18 23:18:12,301 INFO mapreduce.Job: Running job: job_1742310641710_0002 2025-03-18 23:18:19,456 INFO mapreduce.Job: Job job_1742310641710_0002 running in uber mode : false 2025-03-18 23:18:19,457 INFO mapreduce.Job: map 0% reduce 0% 2025-03-18 23:18:24,551 INFO mapreduce.Job: map 100% reduce 0% 2025-03-18 23:18:29,602 INFO mapreduce.Job: map 100% reduce 100% 2025-03-18 23:18:30,617 INFO mapreduce.Job: Job job_1742310641710_0002 completed successfully 2025-03-18 23:18:30,703 INFO mapreduce.Job: Counters: 54 File System Counters FILE: Number of bytes read=43 FILE: Number of bytes written=552145 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=113 HDFS: Number of bytes written=25 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5490 Total time spent by all reduces in occupied slots (ms)=4870 Total time spent by all map tasks (ms)=2745 Total time spent by all reduce tasks (ms)=2435 Total vcore-milliseconds taken by all map tasks=2745 Total vcore-milliseconds taken by all reduce tasks=2435 Total megabyte-milliseconds taken by all map tasks=2810880 Total megabyte-milliseconds taken by all reduce tasks=2493440 Map-Reduce Framework Map input records=2 Map output records=4 Map output bytes=41 Map output materialized bytes=43 Input split bytes=88 Combine input records=4 Combine output records=3 Reduce input groups=3 Reduce shuffle bytes=43 Reduce input records=3 Reduce output records=3 Spilled Records=6 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=100 CPU time spent (ms)=1470 Physical memory (bytes) snapshot=524570624 Virtual memory (bytes) snapshot=5171003392 Total committed heap usage (bytes)=391643136 Peak Map Physical memory (bytes)=300306432 Peak Map Virtual memory (bytes)=2581856256 Peak Reduce Physical memory (bytes)=224264192 Peak Reduce Virtual memory (bytes)=2589147136 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=25 File Output Format Counters Bytes Written=25 [liang@node2 ~]$
查看结果
[liang@node2 ~]$ hdfs dfs -cat /out/part-r-00000 hadoop 1 hello 2 world 1
集群实用脚本
编写脚本一般步骤：
1.在node2的~/bin目录下创建脚本
2.给脚本添加执行权限
chmod +x ~/bin/<脚本名称>
3.测试
统一执行jps脚本
jpsall
vim ~/bin/jpsall
内容如下
#!/bin/bash
for host in node2 node3 node4
do
echo =============== $host ===============
ssh $host jps
done添加执行权限
chmod +x ~/bin/jpsall
测试
jpsall
hadoop启停脚本
hdp.sh
vim ~/bin/hdp.sh
内容如下
#!/bin/bash
if [ $# -lt 1 ]
then
echo "No Args Input..."
exit ;
fi
case $1 in
"start")
echo " =================== 启动 hadoop集群 ==================="
echo " --------------- 启动 hdfs ---------------"
ssh node2 "/opt/module/hadoop-3.3.4/sbin/start-dfs.sh"
echo " --------------- 启动 yarn ---------------"
ssh node3 "/opt/module/hadoop-3.3.4/sbin/start-yarn.sh"
echo " --------------- 启动 historyserver ---------------"
ssh node2 "/opt/module/hadoop-3.3.4/bin/mapred --daemon start historyserver"
;;
"stop")
echo " =================== 关闭 hadoop集群 ==================="
echo " --------------- 关闭 historyserver ---------------"
ssh node2 "/opt/module/hadoop-3.3.4/bin/mapred --daemon stop historyserver"
echo " --------------- 关闭 yarn ---------------"
ssh node3 "/opt/module/hadoop-3.3.4/sbin/stop-yarn.sh"
echo " --------------- 关闭 hdfs ---------------"
ssh node2 "/opt/module/hadoop-3.3.4/sbin/stop-dfs.sh"
;;
*)
echo "Input Args Error..."
;;
esac添加执行权限
chmod +x ~/bin/hdp.sh
测试
hdp.sh start hdp.sh stop
集群机器执行相同命令脚本
same.sh
vim ~/bin/same.sh
内容如下
#!/bin/bash
# 1.获取参数个数，小于1个参数报错
if [ $# -lt 1 ]
then
echo "No Args command Input..."
exit ;
fi
# 2.获取当前机器的路径
currDir=$pwd
# 3.ssh到每一台机器，切换到执行脚本机器的当前目录并执行相应命令，这里执行的命令只支持3个参数，可自己根据实际情况扩展，一般用于查看路径或文件内容
for host in node2 node3 node4
do
echo =============== $host ===============
ssh $host "cd $currDir;$1 $2 $3;"
done添加权限
chmod +x ~/bin/same.sh
测试，ls命令查看三台机器的/home目录，命令如下
same.sh ls /home
集群机器一键关机脚本
gj.sh
vim ~/bin/gj.sh
内容如下
#!/bin/bash
for host in node4 node3 node2
do
echo =============== $host ===============
ssh $host "sudo init 0"
done添加权限
chmod +x ~/bin/gj.sh
测试
gj.sh
如有需要，可点击查看：配套视频教程
完成！enjoy it！
扫一扫
567
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
