背景:
sqoop 从hive导数据到mysql,出现异常 java.lang.RuntimeException: Can't parse input data: '\N',For input string: "\N"
Error: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: Can't parse input data: '\N'
at xxx.__loadFromFields(xxx.java:421)
at xxx.parse(t_user_analysis_pv_view_day.java:344)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: "\N"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at xxx.__loadFromFields(xxx.java:403)
... 12 more
原始语句:
sqoop export --connect jdbc:mysql://127.0.0.1:3306/db_test --username test --password '123456' --table test --fields-terminated-by '\t' --update-key date --update-mode allowinsert --export-dir /usr/hive/warehouse/dwd_test.db/test/statdate=20200916 --num-mappers 1 --columns="id, date, pv_view, client_type, entrance_type, module_type";
解决问题:
主要是因为hive 中的表数据为空导致的,添加一个参数忽略空☞即可,--input-null-non-string '\\N'
sqoop export --connect jdbc:mysql://127.0.0.1:3306/db_test --username test --password '123456' --table test --fields-terminated-by '\t' --input-null-non-string '\\N' --update-key date --update-mode allowinsert --export-dir /usr/hive/warehouse/dwd_test.db/test/statdate=20200916 --num-mappers 1 --columns="id, date, pv_view, client_type, entrance_type, module_type";