Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@5707bcf

最新推荐文章于 2022-02-21 08:59:12 发布

qq_43478653

最新推荐文章于 2022-02-21 08:59:12 发布

阅读量853

点赞数 2

本文链接：https://blog.csdn.net/qq_43478653/article/details/112801004

版权

大数据同时被 2 个专栏收录

6 篇文章 0 订阅

订阅专栏

异常记录

4 篇文章 0 订阅

订阅专栏

项目场景：

Windows10、Hadoop2.7.7

问题描述：

2021-01-19 10:24:07,714 INFO [org.apache.hadoop.conf.Configuration.deprecation] - session.id is deprecated. Instead, use dfs.metrics.session-id
2021-01-19 10:24:07,715 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Initializing JVM Metrics with processName=JobTracker, sessionId=
2021-01-19 10:24:08,637 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-19 10:24:08,668 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2021-01-19 10:24:08,743 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 4
2021-01-19 10:24:08,758 INFO [org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat] - DEBUG: Terminated node allocation with : CompletedNodes: 1, size left: 0
2021-01-19 10:24:08,783 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:3
2021-01-19 10:24:08,849 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1526626776_0001
2021-01-19 10:24:09,001 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2021-01-19 10:24:09,002 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1526626776_0001
2021-01-19 10:24:09,004 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2021-01-19 10:24:09,011 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 1
2021-01-19 10:24:09,013 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-01-19 10:24:09,067 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2021-01-19 10:24:09,068 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1526626776_0001_m_000000_0
2021-01-19 10:24:09,088 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 1
2021-01-19 10:24:09,095 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2021-01-19 10:24:09,816 INFO [org.apache.hadoop.mapred.Task] -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@1fee23cf
2021-01-19 10:24:09,822 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: Paths:/user/chinasoft/input/6.8mb.txt:0+3559424,/user/chinasoft/input/6.8mb.txt:3559424+3559424
2021-01-19 10:24:09,875 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2021-01-19 10:24:09,875 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2021-01-19 10:24:09,875 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2021-01-19 10:24:09,875 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2021-01-19 10:24:09,875 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2021-01-19 10:24:09,878 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-19 10:24:09,881 INFO [org.apache.hadoop.mapred.MapTask] - Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@5707bcf4
java.lang.NullPointerException
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:523)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2004)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2021-01-19 10:24:09,883 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2021-01-19 10:24:09,902 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1526626776_0001_m_000001_0
2021-01-19 10:24:09,903 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 1
2021-01-19 10:24:09,903 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2021-01-19 10:24:10,004 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1526626776_0001 running in uber mode : false
2021-01-19 10:24:10,005 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%
2021-01-19 10:24:10,418 INFO [org.apache.hadoop.mapred.Task] -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@432cc287
2021-01-19 10:24:10,421 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: Paths:/user/chinasoft/input/5.1mb.txt:0+2666496,/user/chinasoft/input/5.1mb.txt:2666496+2666496
2021-01-19 10:24:10,463 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2021-01-19 10:24:10,463 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2021-01-19 10:24:10,463 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2021-01-19 10:24:10,463 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2021-01-19 10:24:10,463 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2021-01-19 10:24:10,463 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-19 10:24:10,464 INFO [org.apache.hadoop.mapred.MapTask] - Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@2c65ef2b
java.lang.NullPointerException
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:523)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2004)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2021-01-19 10:24:10,464 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2021-01-19 10:24:10,478 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1526626776_0001_m_000002_0
2021-01-19 10:24:10,479 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 1
2021-01-19 10:24:10,479 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2021-01-19 10:24:11,081 INFO [org.apache.hadoop.mapred.Task] -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4605402b
2021-01-19 10:24:11,085 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: Paths:/user/chinasoft/input/1.7mb.txt:0+1769472,/user/chinasoft/input/3.4mb.txt:0+3555328
2021-01-19 10:24:11,128 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2021-01-19 10:24:11,128 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2021-01-19 10:24:11,128 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2021-01-19 10:24:11,128 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2021-01-19 10:24:11,128 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2021-01-19 10:24:11,129 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-19 10:24:11,129 INFO [org.apache.hadoop.mapred.MapTask] - Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@63988a96
java.lang.NullPointerException
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:523)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2004)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2021-01-19 10:24:11,129 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2021-01-19 10:24:11,144 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2021-01-19 10:24:11,154 WARN [org.apache.hadoop.mapred.LocalJobRunner] - job_local1526626776_0001
java.lang.Exception: java.lang.NullPointerException
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2021-01-19 10:24:12,008 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1526626776_0001 failed with state FAILED due to: NA
2021-01-19 10:24:12,016 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 0

原因分析：

包导入错误。说明

如果您正在使用新的Hadoop API(Hadoop 0.20.2及更高版本),则必须从包org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat中导入KeyValueTextInputFormat.class类,如果您使用的是旧的Hadoop API,你必须从org.apache.hadoop.mapred.KeyValueTextInputFormat导入它

解决方案：

需要更改导入包

import org.apache.hadoop.mapred.lib.CombineTextInputFormat;

为这个包：

import org.apache.hadoop.mapreduce.lib.input.CombineTextInputFormat;

参考博客

https://www.cnblogs.com/bclshuai/p/12329390.html
java – 在Mapreduce中设置job.setInputFormatClass时出错: [http://www.voidcn.com/article/p-gmlfxuzm-bvp.html]

qq_43478653

关注

2
点赞
踩
0

收藏

觉得还不错? 一键收藏
2
评论
Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@5707bcf

项目场景：Windows10、Hadoop2.7.7问题描述：2021-01-19 10:24:07,714 INFO [org.apache.hadoop.conf.Configuration.deprecation] - session.id is deprecated. Instead, use dfs.metrics.session-id2021-01-19 10:24:07,715 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Init
复制链接

扫一扫