解决MapReduce写入数据报Type mismatch in key from map异常

解决MapReduce写入数据报Type mismatch in key from map异常

问题描述

• 自定义MapReduce向hbase写入数据报异常

错误信息

Error: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.hbase.io.ImmutableBytesWritable, received org.apache.hadoop.io.LongWritable
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1081)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:721)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at com.xzlawin.hadoop.StudentMapper.map(StudentMapper.java:16)
	at com.xzlawin.hadoop.StudentMapper.map(StudentMapper.java:9)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

分析原因

• at com.xzlawin.hadoop.StudentMapper.map(StudentMapper.java:16)
• key 和 value 类型匹配错误，原因是没有给Driver设置Map阶段输出类型

// Map阶段不做处理，直接写出到Reducer
context.write(key, value);

解决方法

• 在Driver类中设置Mapper阶段输出数据的key和value类型

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(TableBean.class);