记一次不成功的拉链表

2019-05-10 02:19:37,565 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1556531708937_6923_r_000000_0: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{},"value":{"_col0":"A904A900-AA07-A0C6-907A-CCC60CBA9AB7","_col1":426.02,"_col2":1,"_col3":426.02,"_col4":429.32,"_col5":0.0,"_col6":"FINISHED","_col7":"2019-04-19 12:13:25.0","_col8":"2019-04-19 00:35:19.0","_col9":22,"_col10":"A8092000-A044-46A9-400A-CB74427979C6","_col11":429.32,"_col12":53.05,"_col13":0.0,"_col14":30,"_col15":"转-凤灵珑-20190419-54413","_col16":0.0,"_col17":3.3,"_col18":"2019-04-19","_col19":"FINISHED","_col20":"2019-04-19 00:35:19.0","_col21":"2019-04-19 14:07:35.0","_col22":"EE31263C-D894-45DA-9220-7288CB84CFD8","_col23":"Y","_col24":"DUE_QUIT","_col25":"N","_col26":"2019-05-08","_col27":"9999-12-31","_col28":"2019-05-08"}}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:265)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{},"value":{"_col0":"A904A900-AA07-A0C6-907A-CCC60CBA9AB7","_col1":426.02,"_col2":1,"_col3":426.02,"_col4":429.32,"_col5":0.0,"_col6":"FINISHED","_col7":"2019-04-19 12:13:25.0","_col8":"2019-04-19 00:35:19.0","_col9":22,"_col10":"A8092000-A044-46A9-400A-CB74427979C6","_col11":429.32,"_col12":53.05,"_col13":0.0,"_col14":30,"_col15":"转-凤灵珑-20190419-54413","_col16":0.0,"_col17":3.3,"_col18":"2019-04-19","_col19":"FINISHED","_col20":"2019-04-19 00:35:19.0","_col21":"2019-04-19 14:07:35.0","_col22":"EE31263C-D894-45DA-9220-7288CB84CFD8","_col23":"Y","_col24":"DUE_QUIT","_col25":"N","_col26":"2019-05-08","_col27":"9999-12-31","_col28":"2019-05-08"}}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:253)
	... 7 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant parquet.hadoop.metadata.CompressionCodecName.SNAPPY;
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:527)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:812)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:919)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:666)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
	at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)

 

报错如上，将parquet作为拉链的底层存储，通过insert overwrite的方式进行拉链，分析原因是以为初始数据导入的时候都在初始分区，这个分区的数据很大，做insert overwrite的时候就是将所有的数据全部读出来，然后关联之后再全部写进去，数据量很大，造成数据很大的倾斜，所以，这里将分区表变成了普通表，然后重新执行拉链，搞定