1.配置文件
！注意
内部通信的接口是 10020
对外的接口是 19888
find mapred-site.xml
find mapred-site.xml
[atguigu@hadoop102 hadoop]$ find mapred-site.xml
mapred-site.xml
配置文件添加
[atguigu@hadoop102 hadoop]$ pwd
/opt/module/hadoop-3.1.3/etc/hadoop
[atguigu@hadoop102 hadoop]$ vim mapred-site.xml
<!-- 历史服务器端地址 addhx-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop102:10020</value>
</property>
<!-- 历史服务器 web 端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop102:19888</value>
</property>最终的配置文件
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- 指定 MapReduce 程序运行在 Yarn 上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!--wordcount在页面上运行不出来，只出个temp -->
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>
</property>
<!-- 历史服务器端地址 addhx-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop102:10020</value>
</property>
<!-- 历史服务器 web 端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop102:19888</value>
</property>
</configuration>
2.分发到每个节点 102,103,104
xsync mapred-site.xml
代码
[atguigu@hadoop102 hadoop]$ pwd
/opt/module/hadoop-3.1.3/etc/hadoop
[atguigu@hadoop102 hadoop]$
[atguigu@hadoop102 hadoop]$
[atguigu@hadoop102 hadoop]$ xsync mapred-site.xml
==================== hadoop102 ====================
sending incremental file list
sent 59 bytes received 12 bytes 142.00 bytes/sec
total size is 1665 speedup is 23.45
==================== hadoop103 ====================
sending incremental file list
mapred-site.xml
sent 1071 bytes received 43 bytes 2228.00 bytes/sec
total size is 1665 speedup is 1.49
==================== hadoop104 ====================
sending incremental file list
mapred-site.xml
sent 1071 bytes received 43 bytes 2228.00 bytes/sec
total size is 1665 speedup is 1.49
[atguigu@hadoop102 hadoop]$3.操作
103上 yarn先关闭然后再打开
/opt/module/hadoop-3.1.3
sbin/stop-yarn.sh
sbin/start-yarn.sh
[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$ bin/mapred --daemon start historyserver
[atguigu@hadoop102 hadoop-3.1.3]$
运行成功
jps
4.初始化数据
首先截屏 然后再word中敲打 最后在ue中转化成utf-8
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mkdir /input
mkdir: `/input': Is not a directory
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$ hdfs dfs -rm -r /input
Deleted /input
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -put wcinput/word.txt /input
2024-08-22 23:17:44,066 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$
Hdfs 已经存在了但是在linux界面没有文件夹不知道为啥
点击历史界面
空的什么原因呢
运行日志(空文件)
[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
2024-08-22 23:34:19,157 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.188.103:8032
2024-08-22 23:34:20,185 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/atguigu/.staging/job_1724338953222_0001
2024-08-22 23:34:20,342 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:34:20,574 INFO input.FileInputFormat: Total input files to process : 1
2024-08-22 23:34:20,616 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:34:20,695 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:34:20,748 INFO mapreduce.JobSubmitter: number of splits:1
2024-08-22 23:34:21,103 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:34:21,187 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1724338953222_0001
2024-08-22 23:34:21,187 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-08-22 23:34:21,884 INFO conf.Configuration: resource-types.xml not found
2024-08-22 23:34:21,885 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-08-22 23:34:22,306 INFO impl.YarnClientImpl: Submitted application application_1724338953222_0001
2024-08-22 23:34:22,534 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1724338953222_0001/
2024-08-22 23:34:22,535 INFO mapreduce.Job: Running job: job_1724338953222_0001
2024-08-22 23:34:38,746 INFO mapreduce.Job: Job job_1724338953222_0001 running in uber mode : false
2024-08-22 23:34:38,749 INFO mapreduce.Job: map 0% reduce 0%
2024-08-22 23:34:46,896 INFO mapreduce.Job: Task Id : attempt_1724338953222_0001_m_000000_0, Status : FAILED
[2024-08-22 23:34:45.193]Container [pid=11680,containerID=container_1724338953222_0001_01_000002] is running 302057984B beyond the 'VIRTUAL' memory limit. Current usage: 237.1 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724338953222_0001_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 11690 11680 11680 11680 (java) 485 112 2547171328 60420 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0001/container_1724338953222_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.102 42414 attempt_1724338953222_0001_m_000000_0 2
|- 11680 11679 11680 11680 (bash) 0 0 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0001/container_1724338953222_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.102 42414 attempt_1724338953222_0001_m_000000_0 2 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000002/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000002/stderr
[2024-08-22 23:34:45.240]Container killed on request. Exit code is 143
[2024-08-22 23:34:45.257]Container exited with a non-zero exit code 143.
2024-08-22 23:34:53,035 INFO mapreduce.Job: Task Id : attempt_1724338953222_0001_m_000000_1, Status : FAILED
[2024-08-22 23:34:51.654]Container [pid=15727,containerID=container_1724338953222_0001_01_000003] is running 271043072B beyond the 'VIRTUAL' memory limit. Current usage: 89.2 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724338953222_0001_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 15727 15726 15727 15727 (bash) 1 1 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0001/container_1724338953222_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.102 42414 attempt_1724338953222_0001_m_000000_1 3 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000003/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000003/stderr
|- 15738 15727 15727 15727 (java) 173 287 2516156416 22551 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0001/container_1724338953222_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.102 42414 attempt_1724338953222_0001_m_000000_1 3
[2024-08-22 23:34:51.728]Container killed on request. Exit code is 143
[2024-08-22 23:34:51.734]Container exited with a non-zero exit code 143.
2024-08-22 23:34:59,126 INFO mapreduce.Job: Task Id : attempt_1724338953222_0001_m_000000_2, Status : FAILED
[2024-08-22 23:34:57.767]Container [pid=15768,containerID=container_1724338953222_0001_01_000004] is running 293947904B beyond the 'VIRTUAL' memory limit. Current usage: 113.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724338953222_0001_01_000004 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 15768 15767 15768 15768 (bash) 0 0 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0001/container_1724338953222_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.102 42414 attempt_1724338953222_0001_m_000000_2 4 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000004/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000004/stderr
|- 15778 15768 15768 15768 (java) 344 180 2539061248 28820 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0001/container_1724338953222_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0001/container_1724338953222_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.102 42414 attempt_1724338953222_0001_m_000000_2 4
[2024-08-22 23:34:57.823]Container killed on request. Exit code is 143
[2024-08-22 23:34:57.826]Container exited with a non-zero exit code 143.
2024-08-22 23:35:06,196 INFO mapreduce.Job: map 100% reduce 100%
2024-08-22 23:35:06,214 INFO mapreduce.Job: Job job_1724338953222_0001 failed with state FAILED due to: Task failed task_1724338953222_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
2024-08-22 23:35:06,321 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=4
Killed reduce tasks=1
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=18129
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=18129
Total vcore-milliseconds taken by all map tasks=18129
Total megabyte-milliseconds taken by all map tasks=18564096
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
[atguigu@hadoop102 hadoop-3.1.3]$
运行的时候抓拍到的
5.重新做一遍
页面删除 2个文件夹
重新运行第3次成功了
日志部分(成功)
页面上面删除 /output
删除临时文件
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
2024-08-22 23:47:59,175 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.188.103:8032
2024-08-22 23:47:59,865 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/atguigu/.staging/job_1724338953222_0003
2024-08-22 23:48:00,004 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:48:00,291 INFO input.FileInputFormat: Total input files to process : 1
2024-08-22 23:48:00,323 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:48:00,392 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:48:00,454 INFO mapreduce.JobSubmitter: number of splits:1
2024-08-22 23:48:00,614 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-22 23:48:00,698 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1724338953222_0003
2024-08-22 23:48:00,698 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-08-22 23:48:00,972 INFO conf.Configuration: resource-types.xml not found
2024-08-22 23:48:00,973 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-08-22 23:48:01,101 INFO impl.YarnClientImpl: Submitted application application_1724338953222_0003
2024-08-22 23:48:01,246 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1724338953222_0003/
2024-08-22 23:48:01,247 INFO mapreduce.Job: Running job: job_1724338953222_0003
2024-08-22 23:48:08,634 INFO mapreduce.Job: Job job_1724338953222_0003 running in uber mode : false
2024-08-22 23:48:08,636 INFO mapreduce.Job: map 0% reduce 0%
2024-08-22 23:48:17,783 INFO mapreduce.Job: map 100% reduce 0%
2024-08-22 23:48:23,839 INFO mapreduce.Job: map 100% reduce 100%
2024-08-22 23:48:23,861 INFO mapreduce.Job: Job job_1724338953222_0003 completed successfully
2024-08-22 23:48:24,001 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=64
FILE: Number of bytes written=435569
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=136
HDFS: Number of bytes written=38
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=6270
Total time spent by all reduces in occupied slots (ms)=3591
Total time spent by all map tasks (ms)=6270
Total time spent by all reduce tasks (ms)=3591
Total vcore-milliseconds taken by all map tasks=6270
Total vcore-milliseconds taken by all reduce tasks=3591
Total megabyte-milliseconds taken by all map tasks=6420480
Total megabyte-milliseconds taken by all reduce tasks=3677184
Map-Reduce Framework
Map input records=5
Map output records=7
Map output bytes=63
Map output materialized bytes=64
Input split bytes=101
Combine input records=7
Combine output records=5
Reduce input groups=5
Reduce shuffle bytes=64
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=354
CPU time spent (ms)=4330
Physical memory (bytes) snapshot=495407104
Virtual memory (bytes) snapshot=5131874304
Total committed heap usage (bytes)=454033408
Peak Map Physical memory (bytes)=314241024
Peak Map Virtual memory (bytes)=2562240512
Peak Reduce Physical memory (bytes)=181166080
Peak Reduce Virtual memory (bytes)=2569633792
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=35
File Output Format Counters
Bytes Written=38
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$
拓展
昨天小集群跑了一个晚上
早上突然断网了 集群好像没挂
继续复现昨天的操作
[atguigu@hadoop102 hadoop-3.1.3]$ pwd
/opt/module/hadoop-3.1.3
[atguigu@hadoop102 hadoop-3.1.3]$
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
详细日志
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
2024-08-23 06:07:33,982 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.188.103:8032
2024-08-23 06:07:34,690 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/atguigu/.staging/job_1724338953222_0005
2024-08-23 06:07:34,822 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-23 06:07:35,055 INFO input.FileInputFormat: Total input files to process : 1
2024-08-23 06:07:35,104 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-23 06:07:35,176 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-23 06:07:35,240 INFO mapreduce.JobSubmitter: number of splits:1
2024-08-23 06:07:35,415 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-08-23 06:07:35,485 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1724338953222_0005
2024-08-23 06:07:35,485 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-08-23 06:07:35,721 INFO conf.Configuration: resource-types.xml not found
2024-08-23 06:07:35,722 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-08-23 06:07:35,811 INFO impl.YarnClientImpl: Submitted application application_1724338953222_0005
2024-08-23 06:07:35,920 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1724338953222_0005/
2024-08-23 06:07:35,921 INFO mapreduce.Job: Running job: job_1724338953222_0005
2024-08-23 06:07:49,250 INFO mapreduce.Job: Job job_1724338953222_0005 running in uber mode : false
2024-08-23 06:07:49,252 INFO mapreduce.Job: map 0% reduce 0%
2024-08-23 06:07:56,430 INFO mapreduce.Job: Task Id : attempt_1724338953222_0005_m_000000_0, Status : FAILED
[2024-08-23 06:07:54.710]Container [pid=17568,containerID=container_1724338953222_0005_01_000002] is running 301685248B beyond the 'VIRTUAL' memory limit. Current usage: 149.8 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724338953222_0005_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 17577 17568 17568 17568 (java) 281 395 2546798592 38054 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0005/container_1724338953222_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.104 41800 attempt_1724338953222_0005_m_000000_0 2
|- 17568 17565 17568 17568 (bash) 0 1 9744384 287 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0005/container_1724338953222_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.104 41800 attempt_1724338953222_0005_m_000000_0 2 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000002/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000002/stderr
[2024-08-23 06:07:54.793]Container killed on request. Exit code is 143
[2024-08-23 06:07:54.808]Container exited with a non-zero exit code 143.
2024-08-23 06:08:05,638 INFO mapreduce.Job: Task Id : attempt_1724338953222_0005_m_000000_1, Status : FAILED
[2024-08-23 06:08:03.829]Container [pid=17618,containerID=container_1724338953222_0005_01_000003] is running 303114752B beyond the 'VIRTUAL' memory limit. Current usage: 292.0 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724338953222_0005_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 17627 17618 17618 17618 (java) 403 441 2548228096 74465 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0005/container_1724338953222_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.104 41800 attempt_1724338953222_0005_m_000000_1 3
|- 17618 17616 17618 17618 (bash) 0 0 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0005/container_1724338953222_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.188.104 41800 attempt_1724338953222_0005_m_000000_1 3 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000003/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000003/stderr
[2024-08-23 06:08:03.892]Container killed on request. Exit code is 143
[2024-08-23 06:08:03.894]Container exited with a non-zero exit code 143.
2024-08-23 06:08:11,765 INFO mapreduce.Job: map 100% reduce 0%
2024-08-23 06:08:16,878 INFO mapreduce.Job: Task Id : attempt_1724338953222_0005_r_000000_0, Status : FAILED
[2024-08-23 06:08:15.957]Container [pid=17729,containerID=container_1724338953222_0005_01_000005] is running 319281664B beyond the 'VIRTUAL' memory limit. Current usage: 211.7 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1724338953222_0005_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 17729 17727 17729 17729 (bash) 0 0 9744384 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0005/container_1724338953222_0005_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.188.104 41800 attempt_1724338953222_0005_r_000000_0 5 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000005/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000005/stderr
|- 17738 17729 17729 17729 (java) 475 39 2564395008 53907 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1724338953222_0005/container_1724338953222_0005_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1724338953222_0005/container_1724338953222_0005_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.188.104 41800 attempt_1724338953222_0005_r_000000_0 5
[2024-08-23 06:08:16.042]Container killed on request. Exit code is 143
[2024-08-23 06:08:16.043]Container exited with a non-zero exit code 143.
2024-08-23 06:08:25,026 INFO mapreduce.Job: map 100% reduce 100%
2024-08-23 06:08:25,056 INFO mapreduce.Job: Job job_1724338953222_0005 completed successfully
2024-08-23 06:08:25,213 INFO mapreduce.Job: Counters: 56
File System Counters
FILE: Number of bytes read=64
FILE: Number of bytes written=435569
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=136
HDFS: Number of bytes written=38
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Failed map tasks=2
Failed reduce tasks=1
Launched map tasks=3
Launched reduce tasks=2
Other local map tasks=2
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=16453
Total time spent by all reduces in occupied slots (ms)=7280
Total time spent by all map tasks (ms)=16453
Total time spent by all reduce tasks (ms)=7280
Total vcore-milliseconds taken by all map tasks=16453
Total vcore-milliseconds taken by all reduce tasks=7280
Total megabyte-milliseconds taken by all map tasks=16847872
Total megabyte-milliseconds taken by all reduce tasks=7454720
Map-Reduce Framework
Map input records=5
Map output records=7
Map output bytes=63
Map output materialized bytes=64
Input split bytes=101
Combine input records=7
Combine output records=5
Reduce input groups=5
Reduce shuffle bytes=64
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=115
CPU time spent (ms)=2740
Physical memory (bytes) snapshot=557060096
Virtual memory (bytes) snapshot=5134290944
Total committed heap usage (bytes)=463470592
Peak Map Physical memory (bytes)=311664640
Peak Map Virtual memory (bytes)=2564157440
Peak Reduce Physical memory (bytes)=245395456
Peak Reduce Virtual memory (bytes)=2570133504
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=35
File Output Format Counters
Bytes Written=38
引申话题的原理
参考
https://www.cnblogs.com/liugp/p/16607424.html
https://www.cnblogs.com/liugp/p/16607424.html
692
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
