MapReduce 异常总结（一）

最新推荐文章于 2023-08-01 14:27:09 发布

肖恩ex

最新推荐文章于 2023-08-01 14:27:09 发布

阅读量2.5k

点赞数 1

文章标签： MapReduce 异常

本文链接：https://blog.csdn.net/purple_orange/article/details/95735601

版权

本文总结了MapReduce在执行过程中遇到的序列化异常和自定义类作为key的异常。序列化异常表现为'Initialization of all the collectors failed.'，解决方法是在context.write中正确使用collector.collect。对于自定义类key的异常，主要问题是类没有实现排序规则，需要继承WritableComparator并重写比较方法以确保在shuffle过程中能正确排序。

摘要由CSDN通过智能技术生成

序列化异常

19/06/13 14:11:46 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
19/06/13 14:11:46 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
19/06/13 14:11:47 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/06/13 14:11:47 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
19/06/13 14:11:47 INFO input.FileInputFormat: Total input paths to process : 1
19/06/13 14:11:47 INFO mapreduce.JobSubmitter: number of splits:1
19/06/13 14:11:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1379621938_0001
19/06/13 14:11:47 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/06/13 14:11:47 INFO mapreduce.Job: Running job: job_local1379621938_0001
19/06/13 14:11:47 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/06/13 14:11:47 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/13 14:11:47 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/06/13 14:11:48 INFO mapred.LocalJobRunner: Waiting for map tasks
19/06/13 14:11:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1379621938_0001_m_000000_0
19/06/13 14:11:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/13 14:11:48 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
19/06/13 14:11:48 INFO mapred.Task:  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@50cb6cc3
19/06/13 14:11:48 INFO mapred.MapTask: Processing split: file:/D:/testdata/flow/input/flow.log:0+30880081
19/06/13 14:11:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/13 14:11:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/13 14:11:48 INFO mapred.MapTask: soft limit at 83886080
19/06/13 14:11:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/13 14:11:48 INFO mapred.MapTask: kvstart = 26214396; len