INFO:.........
......
Error while processing statement: FAILED:Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
以上是在运行mapreduce计算时报错代码
问题所在:hadoop配置文件(mapred-site.xml)
#添加
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/opt/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/opt/hadoop</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/opt/hadoop</value>
</property>
#注意修改文件路径
#修改文件后重启集群
sbin/start-dfs.sh
sbin/start-hdfs.sh
*******************mapreduce运行正常检验*******************
在HDFS文件系统中创建文件夹input
hadoop dfs -mkdir -p /input
并上传一份随机文件到HDFS文件系统/input目录下
hadoop dfs -put /root/hivedata/data.txt /input
切换目录到mapreduce的jar包目录
cd /opt/hadoop/share/hadoop/mapreduce/
进入此目录实验运行wordcount mapreduce任务
hadoop jar hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
[root@xxxx01 mapreduce]# hadoop jar hadoop-mapreduce-examples-3.1.3.jar wordcount /input/covid19_data.txt /output
2023-11-06 14:08:16,964 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.137.101:8032
2023-11-06 14:08:17,886 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1698890819796_0010
2023-11-06 14:08:18,041 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-11-06 14:08:18,612 INFO input.FileInputFormat: Total input files to process : 1
2023-11-06 14:08:22,026 INFO impl.YarnClientImpl: Submitted application application_1698890819796_0010
2023-11-06 14:08:22,229 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1698890819796_0010/
2023-11-06 14:08:22,252 INFO mapreduce.Job: Running job: job_1698890819796_0010
2023-11-06 14:08:39,105 INFO mapreduce.Job: Job job_1698890819796_0010 running in uber mode : false
2023-11-06 14:08:39,112 INFO mapreduce.Job: map 0% reduce 0%
2023-11-06 14:08:44,189 INFO mapreduc,280 INFO mapreduce.Job: map 100% reduce 100%
2023-11-06 14:08:51,296 INFO mapreduce.Job: Job job_1698890819796_0010 completed successfully
2023-11-06 14:08:51,430 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=1504
FILE: Number of bytes written=439213
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1413
HDFS: Number of bytes written=1370
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2797
Total time spent by all reduces in occupied slots (ms)=4111
Total time spent by all map tasks (ms)=2797
Total time spent by all reduce tasks (ms)=4111
Total vcore-milliseconds taken by all map tasks=2797
Total vcore-milliseconds taken by all reduce tasks=4111
Total megabyte-milliseconds taken by all map tasks=2864128
Total megabyte-milliseconds taken by all reduce tasks=4209664
Map-Reduce Framework
Map input records=32
Map output records=32
Map output bytes=1434
Map output materialized bytes=1504
Input split bytes=108
Combine input records=32
Combine output records=32
Reduce input groups=32
Reduce shuffle bytes=1504
Reduce input records=32
Reduce output records=32
Spilled Records=64
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=217
CPU time spent (ms)=1450
Physical memory (bytes) snapshot=447250432
Virtual memory (bytes) snapshot=5563158528
Total committed heap usage (bytes)=379060224
Peak Map Physical memory (bytes)=272171008
Peak Map Virtual memory (bytes)=2779856896
Peak Reduce Physical memory (bytes)=175079424
Peak Reduce Virtual memory (bytes)=2783301632
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1305
File Output Format Counters
Bytes Written=1370
出现以上代码说明mapreduce修复成功!!!!!!!