使用StreamSets Data Collector(以下简称sdc)的Hive漂移同步解决方案时,sdc自动增加Hive表字段,发现Hive where查询报错,Impala没问题。
可能的原因:一次性添加字段过多。
解决:在建表时,将大部分或全部的字段创建出来。后续sdc自动增加字段时,就不会出现增加太多的字段。
报错如下:
ERROR : Job Submission failed with exception 'java.util.concurrent.ExecutionException(java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'STRING:STRING:INT:INT:INT:STRING:INT:STRING:INT:STRING:STRING:STRING:INT:INT:STRING:INT:INT:STRING:INT:INT:INT:INT:INT:INT:STRING:STRING:STRING:STRING:STRING:INT' but 'STRING' is found.)'
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'STRING:STRING:INT:INT:INT:STRING:INT:STRING:INT:STRING:STRING:STRING:INT:INT:STRING:INT:INT:STRING:INT:INT:INT:INT:INT:INT:STRING:STRING:STRING:STRING:STRING:INT' but 'STRING' is found.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hive.ql.exec.Utilities.getInputPathsWithPool(Utilities.java:3093)
at org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3054)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:380)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'STRING:STRING:INT:INT:INT:STRING:INT:STRING:INT:STRING:STRING:STRING:INT:INT:STRING:INT:INT:STRING:INT:INT:INT:INT:INT:INT:STRING:STRING:STRING:STRING:STRING:INT' but 'STRING' is found.
at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:372)
at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:355)
at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:416)
at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:329)
at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:818)
at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:124)
at org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat.getHiveRecordWriter(AvroContainerOutputFormat.java:63)
at org.apache.hadoop.hive.ql.exec.Utilities.createEmptyFile(Utilities.java:3150)
at org.apache.hadoop.hive.ql.exec.Utilities.createDummyFileForEmptyPartition(Utilities.java:3181)
at org.apache.hadoop.hive.ql.exec.Utilities.access$300(Utilities.java:213)
at org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3125)
at org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3103)
... 4 more
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'STRING:STRING:INT:INT:INT:STRING:INT:STRING:INT:STRING:STRING:STRING:INT:INT:STRING:INT:INT:STRING:INT:INT:INT:INT:INT:INT:STRING:STRING:STRING:STRING:STRING:INT' but 'STRING' is found.
INFO : Completed executing command(queryId=hive_20200821143211_6cc26546-8c57-43da-8a67-d3d7d1ae185e); Time taken: 0.026 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'STRING:STRING:INT:INT:INT:STRING:INT:STRING:INT:STRING:STRING:STRING:INT:INT:STRING:INT:INT:STRING:INT:INT:INT:INT:INT:INT:STRING:STRING:STRING:STRING:STRING:INT' but 'STRING' is found. (state=08S01,code=1)