sparksql on hive实践

hive-site.xml拷贝到spark的conf目录下后，编程测试如下：

import org.apache.spark._

object exptest {
  def main(args: Array[String]){
    System.setProperty("hadoop.home.dir", "C:\\winutils\\")
    var masterUrl = "local[1]"

    if (args.length == 1){
      masterUrl = args(0)
    }

    val conf = new SparkConf().setMaster(masterUrl).setAppName("Spark-Hive")
    val sc = new SparkContext(conf)

    // sparksql
    val SqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

    SqlContext.sql("show tables").collect().foreach(println)

    sc.stop()

  }
}

打包上传至集群运行

$SPARK_HOME/bin/spark-submit \
  --master yarn-cluster \
  --class com.spark.basic.exptest \
  /home/ubuntu/ispecexp.jar \
  yarn-cluster 

报错日志如下：

 16/12/05 10:15:09 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient)
16/12/05 10:15:09 INFO spark.SparkContext: Invoking stop() from shutdown hook
16/12/05 10:15:09 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}
16/12/05 10:15:09 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null}
16/12/05 10:15:09 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null}
16/12/05 10:15:09 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,nul

可能是没有正常启动Hive的Metastore Server服务进程导致，执行命令hive --service metastore &

重新运行程序，还是有报错：

javax.jdo.JDOFatalUserException: Class org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.

应该是datanucleus-api-jdo-3.2.6.jar，datanucleus-core-3.2.10.jar，datanucleus-rdbms-3.2.9.jar这几个文件没有找到的问题，修改运行命令如下

$SPARK_HOME/bin/spark-submit \
 --master yarn-cluster \
  --class com.spark.basic.exptest \
 --files /usr/local/spark/conf/hive-site.xml \
 --jars /usr/local/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/local/spark/lib/datanucleus-core-3.2.10.jar,/usr/local/spark/lib/datanucleus-rdbms-3.2.9.jar,/usr/local/spark/lib/mysql-connector-java-5.1.37.jar \
 /home/ubuntu/ispecexp.jar \
  yarn-cluster 

程序正常运行。