Hive V1.2.1源码的解译

在利用spark sql on hive的过程中，访问Mysql总是报错，其报错的日志总是显示：

15/09/21 11:12:20 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.   Encountered: "@" (64), after : "".

15/09/21 11:12:20 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

15/09/21 11:12:20 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

15/09/21 11:12:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

15/09/21 11:12:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

15/09/21 11:12:21 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing

15/09/21 11:12:21 INFO ObjectStore: Initialized ObjectStore

15/09/21 11:12:21 INFO HiveMetaStore: Added admin role in metastore

15/09/21 11:12:21 INFO HiveMetaStore: Added public role in metastore

15/09/21 11:12:21 INFO HiveMetaStore: No user is added in admin role, since config is empty

15/09/21 11:12:21 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO Driver: Concurrency mode is disabled, not creating a lock manager

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING)

15/09/21 11:12:21 INFO ParseDriver: Parse Completed

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO SemanticAnalyzer: Starting Semantic Analysis

15/09/21 11:12:21 INFO SemanticAnalyzer: Creating table src position=27

15/09/21 11:12:21 INFO HiveMetaStore: 0: get_table : db=default tbl=src

15/09/21 11:12:21 INFO audit: ugi=ndscbigdata ip=unknown-ip-addr cmd=get_table : db=default tbl=src

15/09/21 11:12:21 INFO HiveMetaStore: 0: get_database: default

15/09/21 11:12:21 INFO audit: ugi=ndscbigdata ip=unknown-ip-addr cmd=get_database: default

15/09/21 11:12:21 INFO Driver: Semantic Analysis Completed

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO Driver: Starting command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING)

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO PerfLogger:

15/09/21 11:12:21 INFO DDLTask: Default to LazySimpleSerDe for table src

15/09/21 11:12:21 INFO HiveMetaStore: 0: create_table: Table(tableName:src, dbName:default, owner:ndscbigdata, createTime:1442805141, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:key, type:int, comment:null), FieldSchema(name:value, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputF ormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMa ps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

15/09/21 11:12:21 INFO audit: ugi=ndscbigdata ip=unknown-ip-addr cmd=create_table: Table(tableName:src, dbName:default, owner:ndscbigdata, createTime:1442805141, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:key, type:int, comment:null), FieldSchema(name:value, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputF ormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMa ps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

15/09/21 11:12:21 ERROR RetryingHMSHandler: MetaException(message:file:/user/hive/warehouse/src is not a directory or unable to create one)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1239)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1294)

at sun.reflect.NativeMethodAccessorImpl .invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl .invoke(NativeMethodAccessorImpl .java:62)

at sun.reflect.DelegatingMethodAccessor Impl.invoke(DelegatingMethodAccessor Impl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)

at com.sun.proxy.$Proxy21.create_table_with_environment_context(Unknown Source)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:558)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:547)

at sun.reflect.NativeMethodAccessorImpl .invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl .invoke(NativeMethodAccessorImpl .java:62)

at sun.reflect.DelegatingMethodAccessor Impl.invoke(DelegatingMethodAccessor Impl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)

at com.sun.proxy.$Proxy22.createTable(Unknown Source)

at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:613)

at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)

at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)

at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)

at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)

at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)

at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)

at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)

at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:329)

at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:310)

at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:139)

at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:310)

at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:300)

at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:472)

at org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33)

at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)

at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)

at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68)

at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)

at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)

at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)

at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:939)

at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:939)

at org.apache.spark.sql.DataFrame.(DataFrame.scala:144)

at org.apache.spark.sql.DataFrame.(DataFrame.scala:128)

at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)

at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:744)

at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:50)

at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)

由于网上对这一块的东西介绍得总是很少，按照操作也总是无解，于是自己想着先把HIVE源码编译一下，花了半天时间，终于搞定。

编译环境：Eclipse

欢迎交流学习，邮箱号：sparkexpert@sina.com