intergrate sparkSQL with hive

1.configuration

copy hive-site to {spack_HOME}/conf ,and add two properties:

<!--    <property>
        <name>hive.metastore.local</name>
        <value>true</value>
    </property>
 -->

    <property>    
       <name>hive.metastore.schema.verification</name>    
       <value>false</value>    
        <description>    
        Enforce metastore schema version consistency.    
        True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic    
              schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures    
              proper metastore schema migration. (Default)    
        False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.    
        </description>    
     </property>  

2.细节问题

默认情况下，Hive是使用derby保存metadata元数据，若使用Mysql来保存metadata，在启动Spark-shell是会出现一个异常

Exception in thread "main" Java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.Hadoop.hive.metastore.HiveMetaStoreClient

解决的方法是添加MySQL的连接驱动到

[plain] view plain copy

1. export SPARK_CLASSPATH=$SPARK_CLASSPATH:/hadoop/hive-1.2.1/lib/mysql-connector-java-5.1.26-bin.jar