sqoop从db2抽取数据报错 SQLException in nextKeyValue

问题:

Error: java.io.IOException: SQLException in nextKeyValue
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:266)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.ibm.db2.jcc.am.SqlException: [jcc][t4][1065][12306][4.16.53] 捕获到 java.io.CharConversionException。有关详细信息,请参阅相连接的 Throwable。 ERRORCODE=-4220, SQLSTATE=null
    at com.ibm.db2.jcc.am.fd.a(fd.java:723)
    at com.ibm.db2.jcc.am.fd.a(fd.java:60)
    at com.ibm.db2.jcc.am.fd.a(fd.java:112)
    at com.ibm.db2.jcc.am.jc.a(jc.java:2870)
    at com.ibm.db2.jcc.am.jc.p(jc.java:527)
    at com.ibm.db2.jcc.am.jc.N(jc.java:1563)
    at com.ibm.db2.jcc.am.ResultSet.getStringX(ResultSet.java:1153)
    at com.ibm.db2.jcc.am.ResultSet.getString(ResultSet.java:1128)
    at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
    at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
    at QueryResult.readFields(QueryResult.java:210)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:246)
    ... 12 more
Caused by: java.nio.charset.MalformedInputException: Input length = 67277
    at com.ibm.db2.jcc.am.r.a(r.java:19)
    at com.ibm.db2.jcc.am.jc.a(jc.java:2862)
    ... 20 more
Caused by: sun.io.MalformedInputException
    at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:105)
    at com.ibm.db2.jcc.am.r.a(r.java:16)
    ... 21 more

原因:
jcc驱动不支持Utf8,需要特殊设置解析字体格式。
When an application uses the IBM Data Server Driver for JDBC and SQLJ (also known as the JCC driver) and is connected to a database with code set UTF-8 (code page 1208), it throws an
SqlException with message including “Caught java.io.CharConversionException” and ERRORCODE=-4220 if the data in a character column that it queries contains a sequence of bytes that is not a valid UTF-8 string.

解决办法
在hadoop的mapred-site.xml中加入下列语句

<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx1024m -Ddb2.jcc.charsetDecoderEncoder=3</value>
</property>

  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值