sparksql与hive整合

最新推荐文章于 2025-03-10 18:42:48 发布

stark_summer

最新推荐文章于 2025-03-10 18:42:48 发布

阅读量3.9w

点赞数

分类专栏： hive spark 文章标签： hive thrift sparksql yarn

本文链接：https://blog.csdn.net/stark_summer/article/details/48443147

版权

hive配置

编辑 $HIVE_HOME/conf/hive-site.xml,增加如下内容:

<property>
  <name>hive.metastore.uris</name>
  <value>thrift://master:9083</value>
  <description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>

启动hive metastore

 启动 metastore:
 $hive --service metastore &
 查看 metastore:
 $jobs
[1]+  Running                 hive --service metastore &

关闭 metastore:

$kill %1

kill %jobid,1代表job id

spark配置

将 $HIVE_HOME/conf/hive-site.xml copy或者软链 到 $SPARK_HOME/conf/
将 $HIVE_HOME/lib/mysql-connector-java-5.1.12.jar copy或者软链到$SPARK_HOME/lib/
copy或者软链$SPARK_HOME/lib/ 是方便spark standalone模式使用

启动spark-sql

standalone模式

./bin/spark-sql --master spark:master:7077 --jars /home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar

yarn-client模式

$./bin/spark-sql --master yarn-client --jars /home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar

执行 sql:
select count(*) from o2o_app;

结果:
302
Time taken: 0.828 seconds, Fetched 1 row(s)
2015-09-14 18:27:43,158 INFO  [main] CliDriver (SessionState.java:printInfo(536)) - Time taken: 0.828 seconds, Fetched 1 row(s)
spark-sql> 2015-09-14 18:27:43,160 INFO  [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - Finished stage: org.apache.spark.scheduler<